The importance of unit testing & version control for scientific software

This post is inspired by the fact that I’m teaching software carpentry in Denmark this week, but I have had this conversation a few times, so I thought I should write it down.

Often the reaction to teaching things like version control or unit testing to scientists is of the sort aren’t these things more appropriate for professional software developers who can put in the effort to learn them? I strongly disagree.

In fact, I’ll defend that unit testing and version control are more important for science than commercial software.

§

Let’s say you are running a web-based business. Unfortunately, your website’s code is a mess. Many of features were implemented by someone who left a while back and none of your new hires really understands what that code does. Fortunately, however, the code works, the site is pleasing to the eye and customers are happily paying for your services. Even these old code bases can have their lives stretched for far longer than you’d expect. Life is not that bad.

Let’s say, on the other hand, you are running a computer-based science enterprise. Unfortunately, your code is a mess. Many of the features were implemented by someone who left a while back and none of your new hires really understands what that code does. Fortunately, the code produces pretty plots. Unfortunately, you cannot explain what the plots represent beyond a vague idea. You can adapt the code to a new dataset, but never really sure why it’s working like it is and sometimes the outputs are downright mysterious. Life is pretty bad. You need to start over.

§

The difference is that in many commercial aspects, only the final output matters. If a website is pretty, it won’t much matter whether the CSS behind it is a mess. If the search engine gives the customers want they want, both costumer and vendor are happy and nobody will say I’ll buy, but first can we go over the methodological details? There are solid reasons to make the code clean and well-tested (in terms of minimizing the negative impact of individual members leaving the team or avoiding increasing costs to maintenance & extension), but it’s not required for success.

In science, however, it is not enough to have a pretty output plot. You also need to be able to explain the details behind the plot and be certain that the plot was produced the way you think it was produced. If the code gives a mysterious result, then it’s not OK to just add a hacky line of code adding 10 to result to make it work. Similarly, the ability to go back in time in your code is a nice thing in business, but can be essential in science because we value reproducibility.

§

This is why I think it’s very good that unit testing & version control are both part of the software carpentry core curriculum.

Advertisement

Why you should learn version control: a personal story

As part of software carpentry instructor training, we were asked to write a motivational story for learning one of the themes of the material. This is mine, which I posted on the software carpentry website:

This is my motivational story to use version control (I do not have yet a story of when I was not motivated to learn in mind, so I will post that one later).

A few years ago, I submitted a paper somewhere and it got accepted. In the meanwhile, a few months had passed and we were now asked to provide the camera ready versions.

In order to generate the higher quality version of the figures, I reran the script which did so, after changing just a few of the output parameters to get a high-resolution version. However, the resulting figure did not look like the one we had submitted! Not so different as to warrant a different conclusion, mind you, but you could see that there was a slight shift in the plots when you looked at them side-by-side.

First, I calmed reviewed the code to see if something obvious popped, then I worriedly reran the whole pipeline to re-generate intermediate results, and finally I started to panic. What was wrong? Had I submitted a paper with a result which I did not know how to reproduce?

Because I keep all my code under version, I rolled back the code to the version which had been available at the time of submission. Now, it regenerated the figure exactly as we had submitted it. Relief.

Using binary search (which git has built in), I was able to isolate the exact code change which caused the difference. It turned out to be a very minor change in the way that a certain computation is made, which was mathematically equivalent but not numerically equivalent (i.e., it would have been the same if computers had infinite precision, but because we round, we obtain different results). This meant that an almost arbitrary decision at one point of the algorithm was done differently and then the results shifted enough to be visible.

Thus, because of version control, I was both able to (1) reproduce the figure at the necessary higher resolution and (2) understand why the results had changed. There was much rejoicing.

 

Putting My Effort Where My Mouth Is

Yesterday, I argued that computer programming should be part of basic instruction.

Today, I’ll talk about a particular group of people who need to learn to programme: scientists, even lab scientists.

Personally, one of the great draws of moving away from academia into industry (the good industry places, at least) is to be able to use decent tools. Code sharing is a solved problem (the solution is version control). CVS was released in 1986. That’s almost 30 years ago. In academia, use of this type of tool is not (yet?) widespread.

§

In this context, I have (for a long time now) been putting my effort where my mouth is and created a course called Programming for Scientists, which I have taught a couple of times.

Last week, at EMBL, I helped teach a course on Python. This was done on a tight schedule and still we were overwhelmed with demand (we sold out in about 24 after a single email announcement).

Software Carpentry is another great project to teach scientists about programming and computer usage.