Go here to sign up for The Embedded Muse.

Embedded Muse 189 - Feb. 1, 2010

You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go to https://www.ganssle.com/tem-subunsub.html or drop Jack an email at jack@ganssle.com.

EDITOR: Jack Ganssle, jack@ganssle.com

Editor's Notes

Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it's not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm .

Quotes and Thoughts

There are various reasons that software tends to be unwieldy, but a primary one is what I like to call "brittleness". Software breaks before it bends, so it demands perfection in a universe that prefers statistics. - Jaron Lanier

Toyota Brakes

Over the last few days I've received an avalanche of email from engineers asking for my take on the "software problem" in certain models of Toyotas that leads to runaway cars. The blogosphere is full of speculation about the problem as well, so much that Toyota's software seems to stand indicted, convicted and tried.

I drive a Toyota Prius, a hybrid that has a regenerative braking system. I'm told the brake pedal is just an input to the computer with no connection to the hydraulic system. After 102,000 miles it has never failed to stop the car, but does feel very subtly "weird" very occasionally, like there's just something different, something unexpected in the feel of the braking system. But it has never felt unsafe.

Engineers who design ABS brake systems tell me the software is qualified by highly experienced test drivers who zoom around the track and come back asking for a different "feel," which is something that can't be embodied in formal requirements. It seems software-controlled braking is designed to model the century-old manual controls, to give the driver the experience he is used to.

My take on the current recall is: we know nothing. Software, floor mats, mechanical problems - it could be one of these, all of them, or something else entirely. When the NTSB sets off to investigate an airplane crash they leave DC with the most important tool that an investigator possesses: an open mind. As inquiring engineers it's fun to speculate about potential root causes of accidents, but that's simple speculation, nothing more.

I do predict that if the software is involved, the engineering community will learn nothing useful from this experience. Surely details of the bug will remain shrouded in secrecy. My hope is that someday we open the books on bugs so, just as is true in civil engineering and the aviation industry, others can learn from our mistakes. Alas, for now that hope seems completely na‹ve.

Geoff Patch had some comments about compilers: "Your item about incorrect processing of the volatile keyword was very interesting. My take on the subject is that this simply reveals the larger problem of almost universally defective compiler optimization technology. The optimiser guys try too hard, and they screw up too often.

"In my experience, improved performance is generally to be found in better data structures and better algorithms. If a project is relying on the optimiser to wring out the last drops of performance required, then that project is in trouble. Compiler optimisers provide relatively small performance increases in most cases, while *frequently* introducing defects that I would categorise as The Work Of Satan. When your compiler is generating defective code, then all bets are off, and your normal debugging techniques go out the door because your source code no longer maps to what the machine is actually executing.

"For example in 1998 I was using a compiler from a well known DSP manufacturer and my application was doing one of those "That's impossible, it simply can't be happening!" things. As a last resort I started reading the compiler generated assembler, and Lo and Behold, the compiler had decided that an entire function, including accesses to volatile variables, wasn't necessary. It simply hadn't generated the code to call the function. Reducing the optimisation level fixed that.

"Maybe it makes me sound like a Luddite, but that episode and others has taught me that the safest course, if at all possible, is to turn off optimisation entirely. Then, if optimisation is required (and it usually isn't), increase optimisation slowly and regression test *everything*."

A New Paradigm?

Howard Smith compares hardware to software engineering: "The Hardware types have figured things out. They can design a microprocessor and get 731 Million transistors in an Intel I7 to work in glorious harmony. Project cycles may be as short as 12 to 18 months. Their goal is 100% functionality on first silicon, and they do achieve this. They have evolved their process, and it is working very well.

"Maybe this is a radical suggestion, but I think it may have some merit. Why not give the software projects to the hardware types? They will figure out a process that works just as well as their hardware process does. They also don't have any problem investing in the tools to get it done. Remember, the software manager balks at approving a $99 compiler, but the hardware manager will approve several full seats of Mentor Graphics tools at upwards of $250,000 each. He will also include the latest digital scope and a logic analyzer, to boot. If they miss the 100% functionality on first silicon, the penalty is a second mask set, and the time to do the second spin of the silicon. That probably is $5M to $10M for the mask set, and a delay of 3 to 6 months. That's a pretty severe penalty. Imagine the cost and time if a software bug resulted in a mandatory upgrade of every device that was ever shipped. That might be as severe, but I doubt that this has ever happened. Investing in excellent development tools is really cheap insurance. Besides, it helps get the job done right, in the first place.

"I really do agree that the problem with software is in the process. The software community seems to be really stuck on the old saying, 'But we've always done it this way.' There has to be somebody who can create the next software paradigm that cures most of our problems. And, it probably will not be a software type that does it."

Tools and Tips

Tim McDonough likes Geany: "I recently learned about an open source programmer's editor called "Geany" that I like very much so far. It has pre-compiled versions for both Windows and Linux environments and has a lot of nice features for managing software projects, build tools, etc. http://www.geany.org/."

Vlad Zeylikman looked at the tool list and found one of his favorites mis-categorized: "Understand (http://www.scitools.com/products/understand/). It's not cheap. the lowest priced version supports C, C++, C#, and Java. Other variants support more languages, including VHDL. It can generate code metrics, call and called trees, cross-references, etc. The code navigation within a project is very easy. Search can be done on project files or outside. There are Windows, Linux, and Solaris versions. For a large project on which you need to come up to speed quickly, it is an invaluable tool. It is way above SourceNavigator's capabilities but then SourceNavigator is free and Understand is not.

"I found Understand on the list but in the wrong section: static analysis tools. It really isn't. It's a source navigator with some analysis capabilities: cyclomatic complexity is the main thing (http://www.scitools.com/products/understand/features.php)."

Joke for the Week

Kinney Bacon sent a number of fun words from the Washington Post's Style Invitational, which once again asked readers to take
any word from the dictionary, alter it by adding, subtracting, or
changing one letter, and supply a new definition. Here are some that we may find too close to home:

Bozone (n.): The substance surrounding stupid people that stops bright ideas from penetrating "The bozone Layer." Unfortunately, shows little sign of breaking down in the near future.

Inoculatte: To take coffee intravenously when you are running late.

Decafalon (n.): The grueling event of getting through the day consuming only things that are good for you.

Dopeler effect: The tendency of stupid ideas to seem smarter when they come at you rapidly.

About The Embedded Muse

The Embedded Muse is a newsletter sent via email by Jack Ganssle. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.