Interactive Design

Build your system incrementally; testing and development is much easier.

By Jack Ganssle

In looking for another firmware engineer I recently had the occasion to interview a new college graduate. It was eerie, really, a bit like dealing with a programmed cult member unthinkingly chanting the cult's mantra. In this case, though, it was the tenants of structured programming mindlessly flowing from his lips.

 It struck me that programming has evolved from a chaotic "make it work no matter what" level of anarchy to a science, or maybe even an "art" whose precepts are practiced without question. yes">  Problem Analysis, Top-Down Decomposition, Partitioning - all of these and more are the N commandments of structured design, commandments we're instructed to follow lest we suffer the pain of (gasp!) normal">failure.

 Occasionally we hear from the heretics. Some OOP advocates suggest cranking some code, fiddling with it a bit to see how the classes fit into your overall system scheme, and then changing things as needed to make it all work. 

There's nothing worse than building a system you don't really understand. It's like creating a city, starting with the spires of the churches and working down.

 And yet! surely there's room for iconoclastic ideas. I fear we've accepted structured design, and all it implies, as a bedrock of our civilization, one buried so deep we never dare to wonder if it's only a part of the solution.

 If programming were a discipline practiced by rigidly - and perhaps mindlessly - following a cookbook, then our salaries might be a lot less. Presumably we're paid for our creativity, for our ability to invent new solutions, and new ways of finding solutions.

 "Design the system, write all the code, and then spend a few minutes debugging" is a nice idea, but one that rarely works. The debug phase always lasts far longer than planned, and becomes the place where heroic actions save a project.

 Debugging takes too long for a long list of reasons I don't care to explore here, all too many related to our limits as people. Probably Lt. Commander Data or Mr. Spock could write perfect code every time, first time. The rest of us struggle with bugs.

 We do know that some singularly effective techniques can drastically reduce debugging. Code walkthroughs, for one.

 But let's be honest. Embedded systems suffer from unusual challenges that no walkthrough can surmount. Embedded systems are not like accounting or database applications. Three problems unique to our field create no end of trouble, and thwart many of our efforts to design first, code second.

 The first is the real time nature of much of our code. It operates in the time domain, as well as in the procedural domain. Much too often we have little idea how the code performs in time until it's written and more or less working. There's plenty we can do to design things to be fast, but fast, normal">fast is still a very qualitative term till the compiler has done it's thing and we're executing firmware on the target system. Only then, in the debug phase, do faces grow white with panic and tempers flare as the magnitude of speed problems surface, sometimes just weeks from promised delivery.

The second is size. Despite our grousing about code bloat, it makes little difference whether a word processor is 10 Mb or 12 Mb. Yeah, it's annoying to have to add a couple of Gb of disk to your computer every year or two, but the PC industry thrives despite the staggering size of even simple programs.

 Embedded systems are generally constrained by their hardware. Grow past the size of your ROM, and suddenly there are serious repercussions. The extra memory costs can quite literally doom a product - and perhaps a company - to failure.

 comp.arch.embedded recently carried a thread about the size of a hello.c program in different compilers, with results ranging from 4k to pretty much infinity. To me, the fascinating undercurrent of this thread is people's surprise, angst, and confusion about the implications of the compilers' outputs. Clearly, size is unpredictable until you compile and check some code.

 The third wringer in the morass of embedded programming is that of the hardware. Let's face it - few of us really understand how all those peripherals work till we play with them. The problem is exacerbated by increasing complexity of the chips we use, and the growing wall between hardware and firmware people.

 If you can't speak the hardware lingo, working with a part that has 100 "easy-to-setup" registers will be impossible. If you are a hardware expert, dealing with these complex parts is merely a nightmare. Count on agony when the databook for a lousy timer weighs a couple of pounds.

 Diddling

This is not a rant against software methodologies - far from it. I think, though, a clever designer will identify risk areas and take steps to mitigate those risks early in a development program. Sometimes cranking code, maybe even lousy code, and diddling with it is the only way to figure out how to efficiently move forward.

 Don't assume you are smart enough to create complex hardware drivers correctly the first time. Plan for problems instead of switching on the usual panic mode at debug time.

 Before writing code, before playing with the hardware, build a shell of an executable using the tools allocated for the project. Use the same compiler, locator (if any), linker, and startup code. Create the simplest of programs, nothing more than the startup code and a null loop in main() (or it's equivalent when working in another language).

 If the processor has internal chip-selects, figure out how to program these and include the setups in your startup code. Then, make the null loop work. This gives you confidence in the system's skeleton, and more importantly creates a backbone to plug test code into.

 Next, create a single, operating, interrupt service routine. You're going to have to do this sooner or later anyway; swallow the bitter pill up front.

 Identify every hardware device your code interfaces to make it operate. Make a list, check it twice - LEDs, displays, timers, serial channels, DMA, communications controllers - include each component.

 Surely you'll use a driver for each, though in some cases the driver may be segmented into several hunks of code, such as a couple of ISRs, a queue handler, and the like.

 Next, set up a test environment for fiddling with the hardware. Use an emulator, a ROM monitor, or any tool that lets you start and stop the code, and manually exercise the ports (issue inputs and outputs to the device).

 Gain mastery of each component by making it do something. It's best to avoid writing code at this point - use your tool's input/output commands. If the port is a stack of LEDs, figure out how to toggle each one on and off. It's kind of fun, actually, to watch your machinations effect the hardware!

 Using a serial port? Connect a terminal and learn how to transmit a single character. Again, manually set up the registers (carefully documenting what you did), using parameters extracted from the databook, using the tool's output command to send characters. Lots of things can go wrong with something as complicated as a UART, so I like to instrument its output with a scope. If the baud rate is incorrect a terminal will merely display scrambled garbage; the scope will clearly show the problem.

 Then write a shell of a driver in the selected language. Take the information gleaned from the databook and proven in your experiments to work, and codify it in code for once and for all. Test the driver. Get it right! You successfully created a module that handles that hardware device. 

Master one portion of a device at a time. On a UART, for example, figure out how to transmit characters reliably, and document what you did, before moving on to receiving. Segment the problem to keep things simple.

 If only we could live with simple programmed inputs and outputs! Most non-trivial peripherals will operate in an interrupt driven mode. Add ISRs, one at a time, testing each one, for each part of the device. For example, with the UART, completely master interrupt driven transmission before moving on to interrupting reception.

 Again, with each small success immediately create, compile and test code before you've forgotten the tricks required to make the little beast operate properly. Databooks are cornucopias of information and misinformation; it's astonishing how often you'll find a bit documented incorrectly. Don't rely on frail memory to preserve this information. Mark up the book, create and test the code, and move on.

 Some devices are simply too complex to yield to manual testing. An ethernet driver, or a IEEE-488 port, both require so much setup that there's no choice but to initially write a lot of code to preset each internal register. These are the most frustrating sorts of devices to handle, as all too often there's little diagnostic feedback - you set a zillion registers, burn a joss stick, and hope it flies.

 If your driver will transfer data using DMA, it still makes sense to first figure out how to use it a byte at a time in a programmed I/O mode. Be lazy - it's just too hard to master the DMA, interrupt completion routines, and the part itself all at once. Get single byte transfers working before opening the Pandora's box of DMA. 

In the "make it work" phase we usually succumb to temptation and hack away at the code, changing bits just to see what happens. The documentation generally suffers. Leave a bit of time before wrapping up each completed routine to tune the comments. It's a lot easier to do this when you still remember what happened and why.

 More than once I've found that the code developed this way is ugly. Downright lousy, in fact, as coding discipline flew out the window during the bit tweaking frenzy. The entire point of this effort is to master the device (first) and create a driver (second). Be willing to toss the code and build a less offensive second iteration. Test that too, before moving on.

Backbone Modules

Your tested library of drivers is one critical part of the code. Before getting terribly serious about the main-line software, fire up significant software components like the RTOS or TCP/IP library.

 You purchased these, right? Except in extreme situations it's rather silly to re-invent the wheel. However, complicated chunks of code rarely drop into place without a battle. Fight it now, before you're trying to simultaneously integrate 50k lines of your own code.

 Create stub tasks or calling routines. Prove to yourself that the module works as advertised and as required. Deal with the vendor, if trouble surfaces, now rather than in a last minute debug panic when they've unexpectedly gone on holiday for a week.

 This is a good time to slip in a ROM monitor, perhaps enabled by a secret command set. It'll come in handy when you least have time to add one - perhaps in a panicked late night debugging session moments before shipping, or for diagnosing problems that creep up in the field.

 Now your system has all of the device drivers in place (tested), ISRs (tested), the startup code (tested) and the major support items like comm packages and the RTOS (again tested). Integration of your own applications code can then proceed in a reasonably orderly manner, plopping modules into a  known-good code framework, facilitating testing at each step.

Human Nature

At the risk of sounding like a new-age romantic, someone working in aroma therapy rather than pushing bits around, we've got to learn to deal with human nature in the design process. Most managers would trade their firstborn for an army of Vulcan programmers, but until the Vulcan economy collapses ("emotionless programmer, will work for peanuts and logical discourse")  we'll have to find ways to efficiently use humans, with all of their limitations.

 We people need a continuous feeling of accomplishment to feel effective, and to be effective. Engineering is all about making things work; it's important to recognize this and create a development strategy that satisfies this need. Lots of little progress points, where we see our system doing something, is tons more satisfying than coding for a year before hitting the ON switch.

 This also puts a sanity factor back into the schedule. Daily, testable, progress is much more realistic than a paper design that goes into a few months of planned debug just before shipment.

 A hundred thousand lines of carefully written and documented code is nothing more than worthless bits until it's tested.. We hear "It's done" all the time in this field, where "done" might mean "vaguely understood" or "coded". To me "done" has one meaning only: "tested". 

Incremental development and testing, especially of the high risk areas like hardware and communications, reduces risks tremendously. Even when we're not honest with each other ("sure, I can crank this puppy out in a week, no sweat"), deep down we usually recognize risk well enough to feel scared. Mastering the complexities up front removes the fear and helps us work confidently and efficiently.

 You'll also sleep better.