Extreme Embedded

By Jack Ganssle

Published in Embedded Systems Programming, January, 2002

Extreme Embedded

Engineering: n. The application of mathematical and scientific principles to practical ends, as the design, construction and operation of economical and efficient structures, equipment and systems. (Webster's II New Riverside University Dictionary, 1984)

So is the phrase "software engineering" an oxymoron?

In the space of a single generation we've created a society intrinsically dependant on software, both of the embedded and non-embedded varieties. No one knows just how much software gets built each year, but it's obvious that every electronic product now has a software component. I often complain about the buggy code that surrounds us, but in fact most software works remarkably well. Despite plenty of software disasters, large and small, the world hums along surprisingly well despite its dependence on programs of all kinds.

But I'd wager that much of what we build is fragile, held together through heroics and Band-Aids. Worse, software ages badly. Entropy sets in, maintenance all too often turns a rickety structure into a house of cards swaying in the virtual breeze. I'm often reminded of an old joke: if architects constructed buildings the way software people make code, the first woodpecker that came along would destroy civilization.

We all know how hard it is to create high-quality software, and the even more difficult process of keeping it clean despite enhancements and maintenance. Like supermodels pursuing the latest fad diet many of us flail about, trying all sorts of new development methodologies and ideas.

Most fail. I think they're usually too big, too complex, too time consuming, and too mired in paperwork. If we decide to use the Capability Maturity Model (CMM), for example, we're guaranteed to fail without major buy-in from the whole company, from the president on down. Since software is generally perceived as a necessary evil, any attempt to rally the organization in a novel but expensive manner is just a high tech form of tilting at windmills.

So I look for small scale approaches, ideas that we developers can adopt individually, or perhaps as small teams. I'd rather fly under the radar screen of the big bosses and make changes now that give immediate benefits. One discipline I adopted years ago, one that is so personal no one even needs to know what you're up to, is the Personal Software Process (see A Discipline for Software Engineering, Watts Humphrey, 1995 Addison-Wesley). Humphrey scaled many of the grand CMM ideas down to individual levels. It's brilliant.

But in recent years there's been a revolution of sorts brewing in the ranks of programmers. Many feel the idea of Big Up-Front Design is flawed. They worry that we're not really smart enough to anticipate all problems on a project, and that the traditional model of doing a detailed design before writing code is wrong. A new philosophy of emphasizing coding over design has spawned several interesting approaches to building products. The most popular, or at least talked about, is eXtreme Programming (XP), a brainchild of Kent Beck.

Frankly, I'm a bit on the fence, feeling that a huge part of the design problem stems from shoddy work, not impossible systems. Most of us detest the tedious effort involved in thinking through the entire design before committing to code. Design is hard; too many of us shy away from the very tough job of doing it right. Management can be even more dysfunctional. When they demand a schedule by Friday, when there's no chance to complete a detailed design, they're in effect telling us to lie to them. The meaningless schedule then dominates the project, shortcutting proper requirements analysis and all the rest.

And yet! even the best-run project is a work-in-process, since requirements invariably change over the course of development. The design is invariably rooted in quicksand. We are wrong to rail against change requests; these are inherently part of a complex effort. (However, when management expects changes without cost they are in the wrong).

Kent Beck envisioned a new software development process where programmers do all of the "good" things in excess. If code reviews are good, we'll do them all of the time. If code that grows convoluted is bad we'll clean it up constantly. If testing is important, then testing will be as important as coding. If customer feedback helps, the customer will be an integral member of the team.

He believes that software projects work best when guided by many, many small course corrections rather than just a few big ones. The philosophy of XP is that everything changes all of the time. The business model changes. Technology evolves. Team members come and go. Requirements change. As he puts it (reference) "The problem isn't change, per se, because change is going to happen; the problem, rather, is the inability to cope with change when  it comes."

Beck derived the XP programming model from four core values, rather than an abstract view of how we should make projects. This approach greatly appeals to me, as I see so many projects head towards disaster unless well-intentioned, courageous developers and leaders keep things on track. His first value is Communication. Problems often come from lack of talk. We don't listen well to the customer, or to Joe down the hall working on a comm interface, or to the boss who is working under constraints we haven't considered.

The second value is simplicity. Keep the code clean. Make it do nothing more than is absolutely required to get the current task done. Gulp. That means don't plan too much for the future; XP believes it's best to do something simple now, and change it later to meet our future, enhanced view of the overall system's need. A fascinating idea, though I waver a bit on this one.

Third, employ massive feedback at each stage of development.  As he says, "Optimism is the occupational hazard of programming. Feedback is the treatment." Brilliant! After all, we know that feedback stabilizes systems!including program development. Feedback means testing constantly to prove (or, at least show) correctness. It means working intimately with the customer.

Finally, we're doomed to failure without Courage, the fourth and perhaps hardest of XP's core values. That means being willing to toss out crummy code and start over. Maybe someone has a crazy idea; courage means spending time investigating it, and being willing to back up and refocus if the idea pans out. Courage means telling the customer - and your boss- the truth about project status.

So how do we translate these values into action? XP defines a dozen practices:

Practice 1: The Planning Game. Business and technical people work together to make basic and critical project decisions. What are the important release dates? How much time will a feature need? important?

The work is divided into "stories", essentially the smallest useful elements of the final system. Negotiations between business and technical people result in a set of stories that forms a release; though each release is much smaller than the entire project.

One difficult part of XP is a willingness to accept a foggy view of the future. The Planning Game does not anticipate a complete project definition, let alone design, since change is so dynamic. This gives me nightmares, though it's interesting and not lightly disregarded.

Practice 2: Small Releases. Get something out to the customer frequently. Aim for releases every few weeks. This practice necessitates an on-site customer who can evaluate the release, suggest or demand changes, or perhaps even accept it.

Many modern programming approaches emphasize frequent releases of subsets of the project. All recognize the difficulty of meeting customer expectations when we dump one huge, completed, system in their laps. The spiral development model (code, release, re-estimate the next iteration, and repeat) is one such method that recognizes the difficulty of up-front estimation, particularly of risky projects.

Practice 3: Metaphor. Just as important as a dreary list of features or stories is an overall, high level description of the project, in terms that the techies, business people and the customer can understand. The words chosen to express the metaphor form the start of a common language used by all of the stakeholders.

Practice 4: Simple Design. XP pushes our quest for simplicity to rather frightening heights. Adherents tell us to write each class or function to do only what the current story demands. Sure, you're thinking that the next story, which you'll start working on in a couple of weeks, might need a more generalized handler. Or maybe we need to abstract this class a little more in case another developer has a specific need. XP admonishes against this. Do what you need now, and change as needed later, to meet future needs.

Philosophically this is one area where XP departs from conventional development models. We're taught that code is very expensive. XP preaches that coding is relatively cheap. Clean, simple code can be reworked for less than we usually figure. Again, a fascinating, though scary, vision.

Practice 5: Testing. Every function has its test suite. The tests are as important as the code; shortchanging tests is strictly forbidden. As we exercise each class or function we build a library of tests that are preserved with the code itself.

This practice ties neatly into the previous one. Fear of changing code diminishes when an extensive test suite is ready to quickly prove or disprove functionality of the modified software.

Though there's certainly nothing novel about stressing the importance of test, to me this is one of XP's most powerful and exciting ideas (OK, so I need to get a life). I've got files of embedded disasters, almost all of which stemmed from lousy testing. Even highly disciplined organizations have difficulty generating adequate tests because these are always constructed based on an external view of what the function is supposed to do; we have a hard time anticipating all of the myriad paths a real function may take to implement a desired capability.

In XP we write the tests as we write the code. This is the time to write a comprehensive test that irons out every possible combination in a five level deep IF. It's a chance to ensure we test each and every exception possibility, as well as the real boundary condition issues that come with passed parameters.

When we change the code later, we change the tests as needed. Neither tests nor production code alone are allowed; both form a synergistic whole.

Testing might be hard to implement with many embedded applications. The XP notion of a test is something rather automated, which may defy our systems that require someone to press buttons and interact with other devices. Yet, how do we dare not create such exhaustive and effective automatic tests?

Practice 6: Refactoring. This is a fancy word for chucking the cruddy code and starting anew. Whenever you see a function that could be simpler, rewrite it. Period. This is a pretty interesting idea, especially when it comes to maintenance, since we know how code bloats.

Practice 7: Pair Programming. In an XP environment two fully-qualified programmers work at a single computer. One implements and types, the other audits. Each takes turns on the hot seat so the pain is shared.

I can hear the howls of protest. This holds little attraction for an anti-social reprobate like myself, but there's plenty to justify the idea. Code Inspections use teams of 4 or 5 people to review the code, and are proven to be up to 20 times more efficient than traditional debugging for finding problems.

XP goes against the mainstream of software engineering thought by coding and reviewing, via the pair programming, all at once. There are no meetings, no slowdowns awaiting an inspection team. It's a very active and dynamic process. I'll admit some reservations here, but do enthusiastically support pair programming over the more usual "Joe retires to the corner to wrestle alone with the function" approach.

Practice 8: Collective Ownership. No one has exclusive ownership of any portion of the code base. If you see a chance to add value to any part of the project, you are required to do so, even if someone else wrote that section. This is another scary idea, since generally no one understands a function like the original author. But the test suite removes much of the risk, since any change is immediately verifiable.

Practice 9: Continuous Integration. Integration is generally a "bad thing", since it defers testing until other components are available. In XP, and other development approaches, we integrate constantly, daily, even hourly. It's the only real way to ensure changes to one section don't break something else. We code and test, test and integrate, relentlessly.

Continuous integration is a superset of the Testing practice. It's role is to ensure that in addition to unit tests we're evaluating the entire system, as it exists at the moment.

Practice 10: 40 Hour Week. Superheroes in the movies never tire, but in real life heroics generally lead to exhausted, error-prone engineers. Yet overtime is not always unavoidable, so the XP rule is you can't work two successive weeks of OT. Cool.

Practice 11: On-Site Customer. The customer doesn't show up once in a while for a demo; he's an integral member of the development team who works, all week long, with these engineers. When questions arise he's there with an answer. At each small release a real live human, one who will use the system, evaluates the project. This is an interactive role, since customers need the developer, not as guides, but as companions. For only when the project actually does something will the customer know what he really wants. The crucible is deployment, real use by real people.

Practice 12: Coding Standards. All code meets an agreed-upon standard. This critical steps makes Collective Ownership possible, since we're using a common style. It eliminates obfuscated software and tricky, confusing constructs.

Summary

In surveys I've done few companies use XP in embedded work. The handful I've talked to, though, all are wildly enthusiastic.

Is XP "the" answer? If course not. It does embody some very powerful ideas.

A wise developer always studies software engineering, stealing at least the best ideas from each new approach. Stasis is death.

Reference: eXtreme Programming Explained, Kent Beck, 2000 Addison-Wesley