Bailing Frantically

By Jack Ganssle

Bailing Frantically

Sadly, business is booming.

In my last real job I developed and sold in-circuit emulators. Talk about a tough product! The designer needs deep insight into every quirk of the target processor, which never works quite as advertised. The firmware is huge and hideously complex, and insane timing requirements make the hardware equally difficult. Just the sort of thing that appeals to geeks like me. But over the years I found an even more fascinating aspect of the business: the customers.

Anyone using a debugging tool is living on the edge of product development. They're in that difficult phase of the project where things just don't work. Typically the schedule is in a state of collapse, the boss's temper flares, and everyone is tired from working too much overtime. In my role as an ICE vendor I worked with thousands of these developers, looking metaphorically or physically over their shoulders as they struggled to complete their projects. I found that few - astonishingly few - follow any sort of even vaguely-disciplined process. Yet the minority who did use reasonable methods outperformed all other developers by astonishing margins. They delivered good code fast. The difference in performance of the two sorts of teams was so staggering I decided to sell the business and become a firmware evangelist.

So now I write and lecture about better ways to create firmware. So many projects are in so much trouble that the demand for this is, sadly, quite high.

It's incredibly satisfying to see how a little bit of information and encouragement can change a development team. But a couple of times a month I hear from a group that can neither attend one of my lectures nor come to an Embedded Systems Conference because they're desperately behind on the latest project. Further inquiry generally shows that the group is always in panic mode. There's no time to learn how better development techniques can bring projects to fruition on-time.

Like sailors on a sinking ship they are too busy bailing to fix the leak. The water slowly rises so they bail ever more frantically. Sooner or later they're going down, but working faster and harder staves of the inevitable end for just a while longer.

An old cartoon shows a fierce battle, the soldiers wielding swords and spears. The general turns away a machine-gun salesman. He complains: "I don't have time to talk to you - can't we see we're fighting?"

The brochure for this year's ESC Boston just arrived. It advertises over 75 classes on all aspects of building embedded systems. There is no other venue that packs so much information into a single week. A press release from conference management suggests that about 5000 attendees are expected. That's a healthy increase over last year, but, in my opinion, an appallingly low number.

We're part of a fraternity of about a quarter million embedded developers. Why is it only 2% of us take advantage of this incredible learning opportunity? Is everyone too busy bailing?

Elusive Perfection

Why are so many firmware projects so late and so bug-ridden? A lot of theories abound about software's complexity and other contributing factors, but I believe the proximate cause is that coding is not something suited to Homo Sapiens. It requires a level of accuracy that is truly super-human. And most of us ain't superman.

Cavemen did not have to bag every gazelle they hunted - just enough to keep from starving. Farmers never expect the entire bag of seeds to sprout; a certain wastage is implicit and accepted. Any merchant providing a service expects to delight most, but not all, customers.

The kid who brings home straight As (not mine, darn it) thrills his parents. Yet we get an A for being 90% correct. Perfection isn't required. Most endeavors in life succeed if we score an A, if we miss our mark by 10% or less.

Except in software. 90% correct is an utter disaster, resulting in an unusable product. 99.9% correct means we're shipping junk. 100K lines of code with 99.9% accuracy suggests some 100 lurking errors. That's not good enough. Software requires near-perfection, which defies the nature of intrinsically error-prone people.

Software is also highly entropic. Anyone can write a perfect 100 line-of-code system, but as the size soars perfection, or near perfection, requires ever-increasing investments of energy. It's as if the bits are wandering cattle trying to bust free from the corral; coding cowboys work harder and harder to avoid strays as the size of the herd grows.

So what's the solution? Is there an answer?

In my opinion, software will always be a problem, and there will never be a silver bullet that satisfies all of the stakeholders. But there are some well-known, though rarely practiced, strategies that offer hope.

The first is to fight featurism. A nifty feature is just a software change, right? We can slap in a bit of extra functionality in no time at all. This attitude has changed cell phones from basically simple devices to million line-of-code firmware monstrosities. The easiest feature to implement is the one that's not there. Prune unneeded requirements to get the code done usually demand far more than any product really needs. We developers usually have little influence over the requirements.

Relentlessly partition the system. Cheat complexity growth by breaking the system up into many small pieces, each of which is as independent as possible. Minimize coupling between functions and modules. As the firmware evolves invariably coupling increases. We add a few more parameters in function calls, sneak in a global or 10, and morph the simple keypad driver into a display/keypad/LED handler. Though I struggle with a lot of what the XP folks mandate, they laudably require that crummy code gets refactored rather than beaten into submission.

Practice reuse. You won't; no one does. But reuse is, in my opinion, our only hope of salvation. Unfortunately, reuse goes against the nature of early 21^st century capitalism, which stresses short term profits uber alles. Reuse fails unless one makes a large up-front effort in generalizing functions. Few bosses accept the argument "if we double development time on this project, we can halve it on the next dozen."

Artistes

Most importantly, use a disciplined development process.

In the olden days all products were hand-made, the work of artistes rather than manufacturers. Plows, shears and guns were all turned out one at a time, each was subtly different than the one the blacksmith finished just the day before. Over time industry learned about the value of making standardized, interchangeable parts, and ultimately about mass production. Though there's long been a retro movement that admires handwork, for most products mass production has yielded far better products at much lower costs.

But not in software, because we view ourselves as artistes, cranking out hand-made code highly optimized for each specific application. Engineering will never resemble a factory, but the utter chaos that rules most firmware shops is unacceptable.

I estimate under 2% of firmware developers have any sort of process that they use religiously. It matters little what that approach may be: XP, CMM, Inspections, Standards, and maybe even Drunken Orgies are all valuable ways to improve the code! when used religiously. None offer much benefit when casually or intermittently employed.

We software artistes have missed the "process" boat. Most other industries use various sorts of defined processes to work efficiently. One way to identify an amateur organization of any sort, be they accountants, lawyers, craft shops or software developers, is by a lack of process. By contrast, an efficient company like MacDonald's defines a rigorous way to do just about everything. Even a teenager, using Ray Kroc's process, can crank out Big Macs that taste exactly as bad as in any MacDonald's in the world.

We firmware folks must look beyond the world of software engineering for insight into better ways to build our products. I highly recommend Michael Gerber's book "The E-Myth Revisited: Why Most Small Businesses Don't Work and What to Do About It" (HarperBusiness, 1995 ISBN 0887307280). Skip the book's irrelevant last half. Gerber says that poor businesspeople work "in" the business - they are technicians daily busy with making the product or service. The business cannot succeed without that individual, who may be a genius at providing some product or service, but spend their days firefighting. He feels the brilliant company owners work "on" the business. They build systems, processes, and techniques so the business runs smoothly. These awesome managers don't just solve problems, they invent solutions that eliminate the problem forever, or that automatically deal with the issue when it comes up again.

They stop bailing and plug the leak.

We must adopt the same philosophy. For instance, I've observed that most teams spend 50% of the project debugging the code. Try something else - maybe pair programming a la eXtreme Programming, or (in my opinion much better) code inspections.

Developers often blame management for lousy development methods. They get neither support nor direction from the boss, who is preoccupied with just shipping the product no matter what it takes. So the developers employ heroics, working ever harder to produce bigger and more complex products.

Nonsense. We cannot abdicate our responsibility to do things right. Another great non-software book is "Quality Is Personal: A Foundation for Total Quality Management" by Harry Roberts (Free Press, 1993, ISBN 0029266254). The author contends that "working harder" is a lousy way to solve problems. The couch potato who promises to watch less TV will probably fail. Better: make a profound change. Throw the TV away, or put it on a timer. Dieting? Empty the fridge, buy only low-cal foods, don't carry money so you don't eat out, but stop making futile promises to do better.

Roberts' philosophy applies to software engineering. Are variable names always totally meaningless? Skip that meeting where everyone promises to do a better job with variables on the next project. It'll fail. Instead, change something. Maybe adopt naming conventions and software standards. Appoint a naming czar. Build a tool that identifies stupid names.

If the boss is indeed capricious and unsupportive, if he thwarts every attempt to improve engineering, you can still change something. Adopt a stealth process, one that requires no buy-in from management, and one you can implement with neither his consent nor knowledge. One that's great is Watts Humphrey's Personal Software Process. The PSP will improve your work even if everyone else on the team is a cowboy.

Conclusion

Most of the developers I talk to do have at least an inkling about how to get their efforts on track. Some understand the potential benefits from using a reasonable process, and others have adopted some or all of these techniques. The vast majority, though, sheepishly admit that they do not employ them religiously.

If the pimple-faced MacDonald's worker doesn't change the deep-frying oil at prescribed intervals, all of the food will be even worse than normal. So the company has a system that insures oil changes happen on-schedule, all of the time. If we don't adopt the same determination to always use a decent process - whatever it is - our code will be a mess. Guaranteed. Cranking C code is, to me, less interesting than finding better ways to crank C.

Stop bailing and plug the leaks.