The Ecosystem of Unix and the Difficulty of Teaching It

Plos One published an awful paper comparing Word vs LaTeX where the task was to copy down a piece of text. Because Word users did better than LaTeX users at this task, the authors conclude that Word is more efficient.

First of all, this fits perfectly with my experience: Word [1] is faster for single page documents, where I don’t care about precise formatting, such as a letter. It says nothing about how it performs on large documents which are edited over months (or years). The typical Word failure mode are “you paste some text here and your image placement is now screwed up seven pages down” or “how do I copy & paste between these two documents without messing up the formatting?” This does not happen so much with a single page document.

Of course, the authors are not happy with the conclusion that Word is better for copying down a short piece of predefined text and instead generalize to “that even experienced LaTeX users may suffer a loss in productivity when LaTeX is used, relative to other document preparation systems.” This is a general failure mode of psychological research: here is a simple, well-defined experimental result in a very artificial setting. Now, let me completely overgeneralize to the real world. The authors of the paper actually say this in their defense: “We understand that people who are not familiar with the experimental methods of psychology (and usability testing) are surprised about our decision to use predefined texts.” That is to say, in our discipline, we are always sort of sloppy, but reviewers in the discipline do the same, so it’s fine.

§

Now, why waste time bashing a Plos One paper in usability research?

Because one interesting aspect of the discussion is that several people have pointed out that Word is better for collaboration because of the Track Changes features. For many of us, this is laughable because one of the large advantages of LaTeX is that you can use version control on the files. You can easily compare the text written today with a version from two months ago, it makes it easier to have multiple people working, &c.[2] In Word, using Track Changes is still “pass the baton” collaboration, whereby you email stuff around and say “now it’s your turn to edit it” [3].

However, this is only valid if you know how to use version control. In that case, it’s clear that using a text-based format is a good idea and it makes collaboration easier. The same way, I actually think that some of the test subjects in the paper had with LaTeX was simply that they did not use an editor with a spell-checker.

The underlying concept is that LaTeX works in an ecosystem of tools working together, which is a concept that we do not, in general, teach people. I have been involved with Software Carpentry and even before that I was trying teach people who are not trained in computers about these sort of tools, but we do not do that great of a job at teaching this concept, of the ecosystem. It is abstract and not directly clear to students why it is useful.

Spending a few hours going through the basic Unix commands seems like a brain-dead activity when people cannot connect this to their other knowledge or pressing needs.

On the other hand, it is very frustrating when somebody comes to me with a problem they have been struggling with for days and, in a minute, I can give them a solution because it’s often “oh, you can grep in extended mode and pipe it to gawk” (or worse, before they finish the description, I’ll say “run dos2unix and it will fix it” or “the problem you are describing is the exact use case of this excellent Python package, so you don’t need to code it from scratch”). Then they ask “how could I learn that? Is there a book/course?” and I just don’t have an answer better than “do this for 10 years and you’ll slowly get it”.

It’s hard to teach the whole ecosystem at once, which means that it’s hard to teach the abstractions behind it. Or maybe, I just have not yet figured out how it would be possible.

§

Finally, let me just remark that LaTeX is a particularly crappy piece of software. It is so incredibly bad that it only survives because the alternatives manage to be even worse. It’s even sadder when you realise that LaTeX is now over 30 years old, while Word is an imitation of even older technology We still have not been able to come up with something that is clearly better.

§

This flawed paper probably had better altmetrics than anything I’ll ever write in science, again showing what a bad idea altmetrics are.

[1] feel free to read “Word or Word-like software” in this and subsequent sentences. I actually often use Google Docs nowadays.
[2] Latexdiff is also pretty helpful in generating diffed versions.
[3] Actually, for collaboration, the Google Docs model is vastly superior as you don’t have to email back-n-forth. It also includes a bit of version control.

IMPACT or How I Learned to Start Worrying and Fear Altmetrics

Altmetrics is the idea that scientific publications should be judged (perhaps primarily) on the impact they have in the general media, including on social media. This is in alternative to looking at either citations of journal impact factors.

§

People who know me outside of science know that antibiotic overuse is a pet peeve of mine [1]. We just published a paper touching on this very subject. It also touched on antibiotic use in agriculture. Both of these can be sold as hot subjects and it’d certainly be possible to try to get some attention in social media with a few bolder statements: antibiotic use in factory farming causes antibiotic resistant infections!

Oh, the altmetrics would go through the roof, but we don’t have the data to support anything like that claim. Our data and analysis is congruent with the idea that antibiotic overuse by humans and farm animals leads to increased resistance which may lead to increased antibiotic resistant infections, but we must acknowledge that there are a large number of confounders and no proof of direct causality. Broadly speaking, people in countries that like to give their animals antibiotics also take a bunch themselves, thus we cannot disentangle farm-to-fork from human antibiotic (over)use. Furthermore, the presence of antibiotic resistant genes is not sufficient to infer the presence of clinically-relevant antibiotic resistant pathogens (this may be a limitation of current methods of analysis, naturally, but a limitation it is). The paper, naturally, has more details on these questions.

We wrote as good scientists, presenting our data and conclusions, acknowledging limitations. We hope to get scientific recognition for this. Most directly in the form of citations, naturally, but more generally in recognition (those people in the Bork lab did a really good job both on their own data and in reviewing other work).

If our incentives were to stir up controversy in social networks, then they would point away from this towards a more polemical stance (and whilst they may, in some sense, draw more engagement with scientific results, they would, in a more fundamental sense, move the discourse away from a evidence-based direction [2]).

When writing blogposts, I put in short pithy sentences for twitter; it’d be dangerous if I did the same when writing a journal paper.

§

Metrics don’t just measure, they also shape behaviour, you need to solve for the equilibrium.

You need to ask: would it be a good thing if people started, on the margin, to optimize for your metric? In the case of scientists and altmetrics, the answer may be NO.

§

An unrelated criticism of altmetrics is that they’d be outright gamed and that the scientific world has nowhere close to the capacity to fight spam like google et al. do. The linked article is also notable for using the word meretricious in the title.

Also, do read the rejoinder.

[1] I’m the sort of guy that when a person complains that their doctor didn’t give them antibiotics for the flu is liable to praise the doctor instead of expressing empathy.
[2] In fact, public diffusion of speculative scientific results can lead to mistrust of science as these speculative results will then tend to contradict themselves leading to dismissal of science in general.

Friday Links

1. A wonderful explanation of Bregman divergences. I also learned that cool result of Banerjee & Gou.

2. I liked The Enduring Myth of the SPARQL Endpoint. This reminded of Titus Brown’ remark that scaling is the only CS problem that still matters.

Money quote: There is a reason there are no ‘SQL Endpoints’.

3. Why proxy measures fail: this applies to Journal Impact Factor, but will also apply to altmetrics. In fact, we can read it as arguing that metrics that are not widely used look better than metrics that are widely used. But it is fallacious to expect that proposed metrics will not go through the same process.

4. President of the AAAS faked her PhD.

5. This battle will eventually reach biology. (This is a much better exposition of the anti-ML position than Chomsky’s rants, by the way).

I think it will be a while before we can just predict any biological behaviour that is interesting, though.

However, how soon before somebody claims that

Every time I fire a biologist, the performance of our systems biology model goes up.

Is the curious task of machine learning to demonstrate to men how little they really know about what they imagine they can study?