Embedded Muse 100 Copyright 2004 TGG August 10, 2004

EDITOR: Jack Ganssle, jack@ganssle.com

CONTENTS:
- Editor’s Notes
- Debouncing
- eXtreme Programming
- Jobs!
- Joke for the Week
- About The Embedded Muse

Editor’s Notes

I’ve created a video titled “Develop Firmware in Half the Time” that distills the basic ideas and processes needed to efficiently crank out great firmware. There’s more information available at https://www.ganssle.com/video.htm.

Civilization fascinates me. How did cities get electricity and telephones? Or gas stations? Imagine the difficulty of tearing up a metropolitan area to add wires, roads, plumbing, gas mains and the like. Fire prevention is another artifact of a maturing society. Once conflagrations that destroyed whole swaths of downtown areas were common. Today technology and fire codes mitigate the perils of blazes. I wrote about the evolution of fire codes and drew some analogies to software engineering for Embedded Systems Programming. Find the article at http://embedded.com/showArticle.jhtml;jsessionid=QYBZP01SXDPQQQSNDBCSKHQ?articleID=26806185.

This is the 100th issue of The Embedded Muse. Muse 1 was sent way back in June of 1997. The Muse evolved out of a similar newsletter (The Embedded Update) I wrote while still in the In-Circuit Emulator business.

Through the last 100 issues readers have sent jokes, ideas, thoughts, criticisms and one or two death threats. The barrage of email is sometimes overwhelming so forgive me if it takes some time to get back to you. But I enjoy the dialog and find it fascinating to hear about people’s projects and processes.

For fun, and to celebrate this not-very-momentous occasion, the Joke for the Week is a 1994 article I wrote for EDN about an embedded sleuth. Find it near the end of this Muse.

All 100 issues are online at https://www.ganssle.com/tem-back.htm.

Debouncing

Check it out at https://www.ganssle.com/debouncing.pdf.

eXtreme Programming

Steve Karg wrote: My first experience with XP was after reading an article in the September 2000 C/C++ Users Journal by Chuck Allison called "The Simplest Automated Unit Test Framework That Could Possibly Work." It included test frameworks written in C, C++, and Java, and opened my eyes to doing best practices to the extreme.

I started reading and studying about XP at that point, and began implementing a few key practices from the XP process: Test First Programming and Refactoring. I had already been doing some of the other key practices, namely Continuous Integration, Coding Standard, 40-hour week, and On-Site Customer.

I started using the Test First Programming and Refactoring on a lighting controller project that I was working on, and found that it really gave me confidence that the code I was writing actually worked correctly. I wrote unit tests for 16 of the 32 modules as I was developing them. The target platform was an embedded PC, so writing unit tests on the PC using Chuck Allison's C framework worked well. Now when I am maintaining that project, it is nothing to go in and Refactor something when I am adding features, and then go back and run the unit tests. I continue to use the Test First Programming method when I can because it changes the way I write the code - I now write it so that it is testable.

I also do My-Process-Subset-of-XP on some of the non-PC based targets, such as code that I write for the MicroChip PIC. However, the code is usually tested and built for the PC first, then ported (although it is usually used as is) to the PIC.

I've tried the Planning Game and Small Releases, but it hasn't fit really well here, and my desk is too cluttered to keep index cards with features on them. I actually use sticky notes with feature requests or bugs and stick them to my monitor.

I haven't really researched using Metaphor.

I started to do Simple Design also, and not adding things that weren't required, but I've found that embedded systems have a bit of an implied list of requirements that are really useful even if they are not required by the design.

Perhaps we are doing Collective Ownership (via the coding standard), but I usually maintain the code I write, and others maintain the code that they write.

We don't do Pair Programming except when working through some tough code that has a bug in it (debugging) and I haven't found the bug by myself after a couple of days of searching. I work in a small shop with just 2 people writing firmware (including me).

Well, I'm exceeding my 40-hours! Hope this helps you with your questions.

A reader who prefers to remain anonymous wrote: Our software group practices XP, and it has been suggested that the embedded group try to implement some of the same practices. Management has not been hard-nosed about it, but it is recognized that improvement in our embedded development cycle is needed.

One of the things that we tried to do was implement the two-week iteration whereby we would try to produce a "deliverable" every two weeks. That time frame seemed to work well on the software side, but it seemed hard to do on the embedded side, particularly when we were trying to bring up hardware and firmware from scratch on a new development -- there just wasn't enough that was stable to make the two week cycle seem feasible.

We tried a four week integration period with mixed success.

One tenet of XP that seems to make sense is developing test cases prior to writing firmware and then writing the firmware to the test case. Automate the test routines to do unit tests against the test cases and wha-lah, you've got very good feedback on whether the latest changes to one part of your code have unwittingly broken other parts of the code.

Unfortunately, most of my embedded development is on an 8-bit 8051 with no OS. The Windows folks have special test suites for their functions to enable the automated execution of unit tests on a nightly basis. I'm still trying to figure out how to do unit testing in an embedded environment, particularly when the firmware has a strong dependency on the hardware, external signals, etc. and where the ability to communicate results may be severely limited.

Rod Chapman and Peter Amey wrote a paper about using XP with SPARK, a subset of Ada that, when used with a static analysis tool, leads to fabulous code. See http://www.praxis-cs.co.uk/pdfs/svandxp.pdf. In it they describe a rather significant variation of XP that Rod tells me as been successful.

A number of folks wrote rather slanderous remarks about XP, but of those it seems none have actually done a real project using it. It seems that eXtreme Programming, especially in the pure form promoted by Kent Beck and others, has little traction in the embedded space (or at least with the 15,000 readers of the Muse).

One wag pointedly asked what development methods I use! There’s some variation depending on the project, but I prefer a merger of Feature Driven Design and Gilb’s Evolutionary approach. FDD is covered in “A Practical Guide to Feature-Driven Development,” by Stephen Palmer and John Felsing (review here: https://www.ganssle.com/books/books2.htm). Evolutionary Development is best covered here: http://www.malotaux.nl/nrm/Evo/

Jobs!

Joke for the Week

They call me Jake in the sleazy end of town, where the streets of the dirty city tell stories of pathos and passion. Edgar Allen Poe lived and died a dirty tramp in a gutter not far away. Old rotting buildings testify to the port's seamier history, a history no amount of urban renewal can remove - nor should it.

"Cool it, Angel. Show the creep in and leave me alone."

Spike stumbled into the old armchair that looked like it went four rounds with an angry bear. A bottle of Captain Morgan fell out of his pocket and shattered on the floor, creating a spicy odor that overwhelmed the scent of mildew.

"It's bad, Jake," he mumbled, scratching the gray stubble on his face. "The Kid just can't make it work. If he doesn't get that thing fixed the customers will off him for sure."

I poured him a shot. He tossed it back, fast, his hands shaking as he slammed the now empty glass on the table. "Jake, this system is killing me. The last set of PC boards was no good. Now we're almost ready to ship it but every few hours the 68000 crashes. I just don't know what we're going to do." As he said this Spike hung his head down, elbows on his knees, a defeated man in a dirty business where the rules change daily and the spec sheets are mostly written in Japanese.

"Tell me more, Spike," I offered, "when does this little bastard crash?"

"Never in the main loop, Jake" he replied. "Only when the command interpreter goes out to service a packet, and then only sporadically. We can execute millions of cycles without a problem, then boom - it's history."

"Ya plan to execute the thing, huh?" I replied... and then realized he didn't mean it quite so literally. "I think it's time we call in Bruno."

"No! Never! I swore I'd never work with him again!" he screamed in a thin whine. "I hate those highly paid consultants! They always make us look so stupid!"

I placed the call. "$4000 a day, plus expenses. In advance," was the gruff reply before the phone slammed down. We waited for his white limo.

Bruno muscled his bulk though the door, a door wide enough for the usual slob but one that seemed pitifully narrow now. Bruno's face was a mass of scars that I knew he had won in a score of battles - the big one on his forehead was from IBM on the ATC project, that slash on his cheek came from the abandoned Sergeant York. Like a Teutonic swordsman's token Heidelberg scar, he wore these proudly. His leather briefcase banged on the floor.

"Hargmpfh!" Bruno was never much for words, but put him in front of a keyboard and his thick fingers were as graceful as Angel doing ballet.

"Well Bruno, you know the score. The system fails once in a while and the engineers have no idea why. It's a new design, but it looks pretty good for an outfit like this."

Bruno's hand crashed down on my desk, making the clip pop out of my .45. "Where's my money?" he rumbled. I scribbled a check which he held to the light for a minute before pocketing it and clumping off down to the lab.

We followed - Spike, slouching and mumbling, me with both hands on my wallet.

The door to the lab creaked open and we walked down the rickety stairs to its floor. Arcs of electricity flowed up Jacob's ladders, while over in one corner a team continued to try and reanimate Elvis's corpse. We ignored the stench and moved to a low slung table with a solitary engineer working feverishly over a small computer system.

"What's dis?" Bruno asked, pointing to a 68000-based factory controller connected via flat ribbon cables to a cage full of STD bus cards.

The Kid stared up at Bruno, fascinated and immobilized by fright, like a deer caught in the headlights of my Packard. His neck let out a sharp click as he looked down again and told Bruno that this was the source of all of his trouble. For three months the Kid had been debugging the hardware and software. He knew he was under the gun - literally - as we had hired him to replace an older engineer who made too much money. I saw the outcast just last week, hustling quarters in the street, and barely recognized him. His sign "can you spare a dime? I know calculus!" gave him away as one of those poor fools who had not learned to quit engineering by age 25.

Trying to be helpful I asked the Kid what he though the problem might be. "I dunno. I've looked at everything. The timing is perfect, the voltage levels are fine. It doesn't make sense!"

With a hand like a side of ham Bruno grabbed the scope probe from the Kid's trembling fingers and started looking at different test points.

"You ain't got no decent gear," Bruno complained. "How am I gonna work with dis lousy 100 MHz scope - it ain't even digital!" I made a quick note to have Spike hit the local test equipment store that night after it closed. Spike would have to be careful, because the cops were starting to get wise to us. They knew there was a pattern to the rash of test equipment capers on the south side.

Bruno pulled out a flip phone and ordered his driver to bring in the 1 GHz four channel beauty I just knew he had stashed in the limo. Then I started listening to Bruno mutterings. At $4000 a day, plus expenses, we had to learn a lot from the big brute.

"Them low frequency scopes with 100 MHz probes can't never really see what's on a high speed bus. I bet dis here sucker is oscillating, like real fast. Real fast," he muttered over and over.

Bruno propped his scope on the back of the liveried chauffeur. He probed the address lines - they looked pretty good to me, tristating as always as a processor enters a bus hold mode. The data lines were awful but data always is a mix of ones, zeroes, and tri-state conditions as the memories decode chip selects and output enables. Only the control lines - read, write, address strobe - seemed to present solid levels all of the time. Still, in a lifetime spent in the grungiest labs in the meanest cities this all looked pretty familiar to me. "What do you think?" I queried.

Bruno's head slowly revolved around to glare at me while the rest of his huge body remained motionless. "Don't rush a expert. I's a master artisan, and I need time to think."

"You have that scope set all wrong," the kid whined, "I always set the vertical channel on 2 v/cm. You've got it on 1 V/cm!"

"Shadup, Kid," Bruno said as he brutally shoved the Kid back onto his stool. "Yoose guys gotta understand that digital circuits are really analog. Ya can't see good enough on a 2V/cm setting to tell if a signal is above the legal minimum one level, or below the legal maximum zero. Hey I betcha don't even know what voltage a logic one is, anyway?"

"CMOS or TTL, HCT or HC?" shot back the Kid.

"Good answer, Kid. Each logic family has its own levels, and ya gotta make sure ya obey the rules. With the scope set to 1V/cm I can see in an instant if any of these levels fall outside of the legal range. Ya know how I hate being illegal. Besides, dat's just the sort of problem dat will cause dis kind of intermittent."

Spike spoke up: "if it's intermittent then maybe it's a broken track or something."

"Yeah. Lemme see." Bruno lifted the board and flexed it gently, all the time watching the monitor. The system kept running. Then he turned the board over and ran his fingers gently over all of the pins. I stared fascinated at the dozens of tiny scars on his fingertips.

"It's dem sharp through-hole leads. Dey cut me up, bad, like Roscoe da Razor did that time in Vegas." Later I learned that Roscoe was in the joint for passing counterfeit 486s. I also learned that Bruno was searching for unconnected input pins that may have drifted to the right state through luck, or 'cause someone up there watches over drunks, sailors, and swill like me.

"Da impedance of any digital output is pretty low. My paws shouldn't mess up the circuit at all," Bruno went on. Despite his almost sensual stroking of each pin the circuit continued to run without fail.

Now Bruno hooked the 1 GHz scope to one data line, and carefully connected the probe's very short ground lead to the IC's ground. He made sure the bandwidth limiter was off, triggered on the read line, and tuned the scope's trigger controls like an old-time ham pulling in a very weak signal. A low growl alerted me that something was up. He checked another data line, and then another.

"Dis ting is oscillating", Bruno roared. "What, yoose got amateurs designing dis stuff now? Watch - I triggered da scope on da read pulse. Look at the data line after da read pulse goes away."

"No one does that," snapped the Kid. I didn't know what Bruno was getting at, but having seen him in a rage once before, I kept mum. I know for a fact that the pub owner now wishes he had too.

Bruno's neck started turning red. I backed slowly away. "Of course da signal's OK during read, you louse. After read, when da bus tristates, it's oscillating at about 450 MHz. Dat'll crash yoose system. Pull-up da data bus with some resistors in a SIP."

"No way, Bruno. The data books only talk about the data bus during read. Who cares what happens during a tri-state condition?"

"Add da SIPs," Bruno demanded. The Kid refused again. Bruno whipped out a 9mm and held it to the Kid's temple. "Da last someone dat messed with my ideas is pushin' up daisies," he rumbled.

I rushed forward to solder the SIPs in myself. If Bruno offed this poor kid the cops might hear the shot.

"Now we go to your office and wait," Bruno intoned, "I've set da system up to run all night to see if it is fixed. Look - the oscillations are gone, and I'll bet you a pair of brass knuckles it's OK now."

For 24 hours we sat, facing each other across the beat up oak conference table, Bruno's eyes boring into mine. He said nothing; I replied in kind. Each twitch escalated into a near shootout as we nervously watched each other's movements, my heater poised under the table, his likewise close at hand. For 24 hours we waited, wondering if this highly paid outsider was worth his $4000 a day, plus expenses.

The new day dawned. The squawk box reported success with the SIPs; a full report was on its way up.

Minutes later, carrying a stack of engineering notebooks, Elvis strode into the room