By Jack Ganssle

Twists on Testing

Published 2/27/2003

Only the clueless build hardware prototypes. Software is always perfect. Big Up Front Design leads to inherently correct systems that need no testing before they're deployed.

Not.

Check is as important as Plan, Do, or Act. Check means run your system against a comprehensive and representative series of tests to make sure the thing does what it's supposed to do.

Kent Beck, the originator of eXtreme Programming, pushes a test-first (http://computer.org/software/homepage/2001/05Design/index.htm) approach. Write the test code before you build the system. Without tests how can you judge your success? How do you really know you're building something that works? Some gurus feel that this is too stringent, and in a sense harks back to Big Up-Front Design, since a test suite implies a complete specification, something the agile folks feel isn't often provided.

No one argues against testing as a critical part of building any system, especially one that uses lots of software.

Except Donald Rumsfeld. The LA Times (http://www.latimes.com/la-na-missile24feb24,1,444539.story) and NPR (http://discover.npr.org/features/feature.jhtml?wfId=1174585) are reporting that the administration wishes to exempt the Missile Defense System (MDS) from a law that mandates extensive tests prior to deployment.

The MDS is viewed by some as an essential shield, by others as an impossible dream, and not a few as a giant handout to the defense industry. Others claim that the current testing regimen is terribly flawed (http://www.ucsusa.org/bmd/bmd_test.html). This isn't the place to debate the merits of such a system. Let's assume it's both necessary and possible.

"Necessary", though, suggests the MDS simply must work reliably. The only tool we engineers have to prove any system functions properly is to put it through its paces. Run comprehensive and realistic experiments designed to uncover design flaws, not tests meant to convince the customer it works.

The administration views testing as an impediment to their aggressive schedule. It probably is. We know, though, that schedules built on false hopes have alienated generations of developers and led to mountains of abandoned systems. A schedule that ignores development realities is always foolish.

I've not seen an estimate of the system's software size, but experts pegged its previous incarnation during the Regan years at 100 million lines of code. There's a lot of room for error in a code base of that size.

Mr. Rumsfeld said he doesn't think we should wait till "every `i' [is] dotted and every `t' crossed" before deploying the system. That's a pretty reasonable statement when building a ship or perhaps even a missile. But in software a single wrong bit can cause complete system collapse. The code will never be perfect, but it's got to be awfully darn good before it will be at all useable.

My take is that if this were a $1 billion system, well, that's chump change (in Federal terms). Skipping tests might make sense since the cost of being wrong would be low. But at $70b we taxpayers should demand a system that works.