By Jack Ganssle

Reliable Firmware

Published 4/19/2001

An Oxymoron?

At April's Embedded System Conference in San Francisco I moderated a "shop talk" - a round-table discussion - on strategies for building firmware. Scheduled for 7:00 AM I expected a half-dozen bleary-eyed attendees, but was delightfully surprised to find the room packed, standing room only, with a lot of passionate and thoughtful engineers.

Is reliable software an oxymoron? Many of the attendees seemed to think so. Several made the point that bugs in shipped products, even in mission-critical devices, are inevitable. One avionics designer said the 747's user manual has an entire section listing software anomalies, with appropriate work-arounds. I started thinking about driving cross-country to the next conference, until realizing that the car is equally chock-full of presumably buggy embedded code.

I shudder when people accept bugs as a fact of life in completed products. The high moral ground says we should ship stuff that works. Period. But. no one knows how to make a large project perfect. Even the Space Shuttle, which is surely the best code ever written in the history of the galaxy (or at least this solar system) has one bug per 400,000 lines. That's in code that costs $1000 per line. Since commercial products run closer to $30/line, there must be a corresponding decrease in quality.

Douglass Adams, the ESC's incredibly entertaining keynote speaker and author of the Hitchhiker's Guide series, claimed in his address that we know when something is "technology" because it doesn't work. It's no longer technology when it fades into the background, when, like a pencil, it's something we don't even think about. Even kids know that when anything electronic acts odd, cycle power.

One wag wrote: the mainframe industry spent 20 years making more fault tolerant computers, while PC companies created more fault-tolerant users. More than a glimmer of truth, there, I'm afraid.

I was amazed, though, that the shop talk attendees all thought that technology wasn't the answer to the quality dilemma. There was a brief mention of OOP as a partial solution, but the conversation quickly moved on.

Everyone - at least all who spoke up - suggested that process and ethics were our only hope of creating good products. I've always thought proper process outweighs any tech tool any time, but had never really thought about the ethics issue before.

By and large we do know how to write damn good firmware. Maybe not perfect, but a lot better than much of the stuff being shipped today. The boss, though, makes rules that compromise quality. When we come up with a schedule based on a capriciously-imposed deadline, we're really lying to ourselves and the boss. Is that ethical?

Our own natural propensity to take shortcuts hurts as well. Aren't we lying when creating a schedule without first doing a detailed design?

Is shipping a known defective product the right thing to do? How much responsibility should individual engineers take?

One attendee made an analogy to the Challenger explosion. Though launching in cold weather was a known hazardous condition, few engineers took a strong stand against the flight. Under pressure from management - that old bugaboo - most of the nay Sayers backed down.

But ethics is never easy. Ethical behavior means taking strong, difficult positions that are often contrary to our own short-term interests. That's tough, especially when most of us take pains to avoid confrontation.

Nancy Reagan suggested teenagers "just say no". Dare we try the same with our boss?

What do you think? Have you ever taken a strong stand against the boss's wishes? What happened?