on software testing

A long time ago, in a galaxy... err, no, scratch that. A long time ago (circa 1999-2000), I used to work for a company which was making software debugging and testing tools. One of the main product was a C++ tool akin to Purify. You'd apply it on your code and it would show you mem leaks, uninitialized pointers, etc... all these pesky bugs which make C/C++ development so enjoyable.

At some point I was working on a GUI in C++, and during an informal meeting with the main boss, he told me that I should be using our C++ tool on my code. I wasn't. Why ? Two reasons. One, because I was reasonably confident about my code. That sounds preposterous and conceited, but I about 99% of the mem allocations in my code where done in a "managed" way. That is, the framework I was using (ok, that was Qt, version 2.x at the time) was handling them for me (yay for object trees).

Upon explaining this to him, he proceeded to show me how wrong I was. He sat down at the keyboard, downloaded the product's last version, untarred it, and, if I recall correctly, first wasted a few minutes trying to install it correctly. Then he tried to apply it on my code. "Applying" meant recompiling my code with the tool, since it was pre-processing the code ('instrumenting' was the exact term) to stuff it with calls to its own libraries before passing it to the actual compiler, then linking with its own libs. I don't recall exactly if we couldn't even get passed that stage, or if the resulting binary was dumping core too quickly to yield any relevant information (if you've used this kind of tool, you know that even system libs are likely to trip it), but it ended up with him finally giving up and concluding with "well, you really should use it".

And that was a perfect demonstration of the 2nd and most important reason why I wasn't using our tool : because it was too darn complicated to use. And from that I shall derive what I think is the single most important feature of a any software testing tool, no matter the language or the framework : ease of use. Using it should be a no-brainer.

Compare the above with a similar tool which we happily use with rosegarden from time to time : valgrind (valgrind appeared about a couple of years after the events above happened). Using valgrind amounts to typing valgrind program-to-check from the command line, and it will present you with a bunch of error reports, most of them relevant (unlike that tool above which would spout a bunch of false alarms). No rebuild, no relink with exotic libs, just run your usual build (compiled with debug info in so you'd get useful stack traces for each error found), and there you go. Sure it runs way slower than usual, but waiting is much less of a problem then spending brain cycles on getting the tool to work.

Software development is hard and complicated. So it's pretty obvious that any tool you use in order to make it less hard and complicated absolutely must not itself be hard and complicated. And yet it seems most such tools completely forget about this, putting cleverness above anything else. "Look at how our tool could find this obscure bug!" Yes, but look at what it took to find it...

So what, you'll say, TANSTAAFL. This is software dev, not walking in the park. True, but the unavoidable consequence is that no matter how effective it is, the tool won't be used. Nobody will use your debugging tool if it takes hours to get working. A regression test suite will soon be left to rot if it's too hard to run and maintain. You can have a team which sole purpose is to bear the brunt of dealing with all test-related issues, but that causes a whole new bunch of problems (communication, training, etc...) A test team should do high-level tests, not functional, fine-grain ones. Those really are for the guy writing the code, and the only way to get him to do them is to make him want to do them. And a developer won't be willing to do something unless it clearly makes his job less complicated.

Why do we use IDEs for Java programming ? Because that squiggly red line under that bit of code I've just typed there telling me there's something wrong is helpful. I don't even have to try building the code to get that error. The gain is obvious. That's how any test tool should be : so simple to use that I will want to use it, without being constrained by some PHB.

Unfortunately there's still no way around the work needed to build a test suite, for instance. But at the very least build it so that running it is as easy as possible (and that's hard too :-) ). Actually in that case, the suite may be based on a test tool, but it becomes a test tool itself so, sorry, it's up to you to make it easy to use (another lesson I learned the hard way).