Toyota's Expensive Software

Summary: Take your pick: do it the expensive way or get it right.

As everyone knows by now, Toyota has agreed to a $1.2 billion fine (http://m.apnews.com/ap/db_289563/contentdetail.htm?contentguid=oQjBROyn) to settle a criminal case over unintended acceleration. A jury in Oklahoma found that, in one case at least, the culprit was the firmware. (The plaintiff's lead expert, Mike Barr, is giving a talk about the case at EELive!).

This payout is on top of another $1B settlement for the same problem (http://www.huffingtonpost.com/2012/12/26/toyota-settlement_n_2366720.html). Bottom line: poor firmware has cost the company staggering amounts of cash.

Testimony shows (http://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences) that the firmware wasn't just suffering from a bug; instead it was reeking with problems. One has to wonder what the engineers were thinking.

Everyone in this industry faces twin pressures: fast and cheap. Deliver now. Cut engineering costs. I have no insight into how many person-hours went into the Toyota code, nor do I know their delivery schedule. But let's look at that most recent $1.2B payout. How does that compare to the engineering effort?

The NASA report (http://www.nhtsa.gov/staticfiles/nvs/pdf/NASA-UA_report.pdf) talks about a code base of "more than 280,000 lines" of code. Mike Barr tells me there were "over a million lines of C source code". For argument's sake, let's figure on a million.

The most expensive code ever written is that of the Space Shuttle, which ran about $1000/LOC (201 Principles of Software Development, Alan M. Davis, 1995). With just the most recent settlement, Toyota's code cost them over $1200 per line - without accounting for any engineering effort. The difference is that the Shuttle's code is the best ever written, averaging about one bug per 400KLOC, and Toyota's has been intensely litigated.

I am not suggesting that Shuttle development practices should be anyone's goal. Perhaps a better benchmark is avionics. It's largely believed that no one has been killed by defective firmware in commercial aircraft, yet that code controls pretty much everything. Sure, the pilots can take over, but modern planes are fly-by-wire. The pilot flies a computer. What does it cost to develop the fabulous software that mediates billions of passenger-miles per day in the air?

Commercial avionics is done to a standard called DO-178B (supplanted recently by DO-178C). Level E applies to software that won't impact operations in any significant way. Level A is for code that can lead to the loss of the aircraft. How much does it cost to write code to level A?

Who knows? Data is sparse and proprietary. However, most pundits figure it's about twice the cost of typical commercial firmware. Others (DO-178B Costs Versus Benefits by Vance Hilderman), in this case based on data from some 150 avionics programs, claim code written to level A is 65% more expensive than that to level E. That figure includes both the engineering effort and the certification process

Note that level C is roughly equivalent to Capability Maturity Model level 3, which is generally considered the entry point to disciplined software engineering. Level A (in the B version of the standard) has 66 objectives that must be met, while there are none at E.

If one skips the certification effort - which is huge - and just writes level A code, some studies (e.g., Safety Critical Software and Development Productivity by O. Benediktsson - though this paper looks at IEC 61508 rather than DO-178B) show that with the use of highly-disciplined processes there is no extra cost to producing the best possible software. Use an ad hoc approach and there's a 70% hit.

Let's be pessimistic and assume the very best avionics costs twice that of typical commercial firmware. My data pegs the latter at $20 to $40 per line of code, from initial specification to shipping. Doubling the high end puts the cost at $80/LOC, or 15 times cheaper than Toyota's most recent payout. Add in their other settlements, legal costs, lost sales, bad PR, and, oh, yeah, the actual firmware engineering, and that difference grows dramatically.

Take your pick: $1200+++/LOC for crappy code, or $80-- for world-class.

One wonders why we as an industry continue to flail around with poor practices that are ultimately expensive and even deadly, when there is a body of knowledge that is provably effective.

Published March 19, 2014