Comments on: The oddities of being a scientist-programmer

By: Rich

Rich — Thu, 19 Mar 2009 14:09:43 +0000

Hi guys, thanks for all the feedback. It’s fascinating to see the range of views there are out there! I agree with Jeffrey, there does very much seem to be a community forming between software engineers and computational scientists/engineers. I think previous generation/s of scientists were able to get away with just regarding a computer as a kind of giant, automated calculator (and act accordingly). It’s not like that any more, because one can do *so* much more with a computer.

By: Jeffrey Carver

Jeffrey Carver — Wed, 18 Mar 2009 14:33:54 +0000

Interesting post. There is a community forming between software engineers and computational scientists and engineers. We are having a workshop at the annual International Conference on Software Engineering. I would really be interested in participation from some of those that read this post. If you are interested, the website is http://www.cs.ua.edu/~SECSE09

By: Terry A. Davis

Terry A. Davis — Wed, 18 Mar 2009 05:04:41 +0000

I once heard a comment, “Is your disertation on physics or Linux?” I have written an operating system for nonprofessional programmers called “LoseThos”, http://www.losethos.com. The good news is I improved the situation for programmers, but the bad news is it’s not standard, so your libraries won’t port very easily. You might check-out the website and offer advice on how it can better fill the niche of scientist programmers.

By: Conrad

Conrad — Tue, 17 Mar 2009 20:47:02 +0000

Beware the “this is just a prototype” mentality. Often today’s proof of concept prototype becomes tomorrow’s legacy system; once features have been added and multiple scientists have made their changes, the code becomes a jumbled mess of indecipherable code.

I’ve found that ensuring that the scientists understand some basic software engineering principles (encapsulation, abstraction and generalization, avoiding copy-and-paste, revision control, use good variable names rather than “d3x”, create recurring unit tests, etc.) and having them pair program with a software engineer now and then really improves the quality of the code.

Once poorly designed code grows to a certain size, making incremental changes to push the science becomes time-consuming, so a little effort in code quality ultimately enables scientists to develop the science faster.

By: Dean Schulze

Dean Schulze — Tue, 17 Mar 2009 17:41:15 +0000

I did so much programming for my PhD level work that when I got a chance to become a software developer I took it. It was a no-brainer since there were no jobs for physics PhDs in the mid-1990s.

It sounds like things haven’t improved very much in the scientific programming field in 15 years. I was struck by how bad – terrible – most scientists code was. The Numerical Recipes family of books is probably the largest collection of bad code in existence.

As an experimentalist I saw many experiments ruined by poor technical work in the lab (bad work on vacuum systems, electronics, etc.). There is lots of bad numerical / statistical work done by scientists too.

In order to “get the science” done the technical work underlying it has to be done correctly.

One PI that I worked for had a full-time staff of two C programmers to write software for data acquisition on his space flight missions. He made no provision for data analysis, however. He left it up to his graduate students (like me). He told me once that “You should be able to just press one key on your workstation and the result should be graphed on the printer.”

I responded that that was true, but only after several man months of programming and proving the code. “Why do you guys think this stuff takes so much time?” he asked. He was very knowledgeable about electronics for space flight and knew that it took man years to write good data acquisition software, but thought that data analysis software was written by elves in the middle of the night.

This was partly driven by funding. He could get multi-million dollar grants to build instrumentation for data acquisition, but NASA and the other funding agencies wouldn’t give a dollar for data analysis. All data analysis software had to be piggy-backed on the hardware funding.

By: links for 2009-03-16 « My Weblog

links for 2009-03-16 « My Weblog — Tue, 17 Mar 2009 04:04:34 +0000

[...] The oddities of being a scientist-programmer | Programming for Scientists (tags: science) [...]

By: Joe Z

Joe Z — Mon, 16 Mar 2009 22:22:31 +0000

The other oddity that always strikes me is that “feature creep” is always a feature of scientific code, rather than a problem. If your code works well then you frequently solve the problem with it and then move on to the general case/slightly different model/new dataset. That (should) make people think more carefully about extensibility when writing their code, and I think some of the most painful codes we have to work with are the ones that have grown organically from a base that wasn’t designed for it.

By: Gustavo Barrancos

Gustavo Barrancos — Mon, 16 Mar 2009 21:33:16 +0000

Totally agree with you Mike. I believe that functional programming languages can be of good use for scientific applications (mathematical expressiveness, numerical tower, automatic memory management etc.)

I wonder what would be the opinion of scientists on languages like Haskell, Clean and Fortress…

By: gawp

gawp — Mon, 16 Mar 2009 21:10:18 +0000

Don’t forget verifying unreliable input sources. That means pretty much anything you didn’t create yourself. Add consistency checks based on how you think the data should act: that will catch both errors in the data and errors in your understanding of the data. I’ve learned this the hard way:

http://www.palidwor.com/blog/?p=33

By: Mike

Mike — Mon, 16 Mar 2009 20:20:26 +0000

That nearly all scientific programming is really prototyping is an excellent and crucial point. An implication of this that is usually missed is that scientific programming should almost always be done using a programming language suitable for prototyping–a language like Python, Ruby, or Lisp. Production languages like C, C++, and FORTRAN should generally be avoided like the plague, used only to write small extensions in cases where the utmost in speed or space efficiency is required.