By Jack Ganssle

The Cost of Quality

Published on embedded.com November, 2012.

Though the paper is over a decade old, in my opinion O. Benediktsson's "Safety Critical Software and Development Productivity" (available here: http://www.eis.mdx.ac.uk/research/SFC/Papers/2WCSQPreprint.pdf) has some stunning results.

We all know that building safety-critical software is hugely expensive. When the code must be utterly reliable the effort expended in getting it right skyrockets. That's why avionics and similar products are so expensive.

Except, that conventional wisdom may not be entirely true.

In the paper Benediktsson looks at IEC 61508, which is a widely-used functional safety standard. It is commonly used (or is sometimes is the father of similar standards) in the automotive. nuclear, rail and other industries. It defines four "safety integrity levels" (SILs) with consequences as follows:

SIL1: Minor injuries at worst
SIL2: Major injuries to one or more persons
SIL3: Loss of a single life
SIL4: Multiple loss of life

The standard lists practices that are recommended, or highly recommended, for each SIL. The use of coding standards, for instance, is highly recommended for every SIL. Formal methods are recommended for SIL2 and 3, and highly recommended for SIL4.

The author relates the effort needed (derived using the COCOMO method) to the difficulty of verifying correctness at various safety levels. Unsurprisingly that goes up by rather a lot as one goes from the lower SILs to higher ones. SIL3 is about 1.7 times more effort than a nominal, non-safety-critical product.

He goes on to relate productivity to process maturity, using the Capability Maturity Model as an example model. At CMM4 he finds schedules are almost half those for nominal products.

Tying it all together results in this table:

In other words, as one progresses to higher levels of process maturity (the CI values - NOM is CMM1, CI3 is CMM4) the effort to build SIL3 systems is no greater than that needed for non-safety-critical systems.

That's a pretty stunning result.

Or, a dysfunctional company can go to high levels of process maturity and get the same crap for half the price.