Go here to sign up for The Embedded Muse.
TEM Logo The Embedded Muse
 Issue Number 262, June 2, 2014
Copyright 2014 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com 		   Jack

You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com.

Contents
Editor's Notes

Adopt best practices and your code will be better and cheaper. This is the thesis of the quality movement, which revolutionized manufacturing but has largely missed software engineering. Studies have even shown that safety-critical code need be no more expensive than the usual stuff if the right processes are followed.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers. Information here shows how your team can benefit by having this seminar presented at your facility.


Tony Gray liked Harley Burton's ruminations in the last Muse and had a recommendation:

I was just reading the career history of Harley Burton in your most recent Embedded Muse, and I thought you (and your readers) might be interested in a terrific book about the heyday of Bell Labs. The title is "The Idea Factory: Bell Labs and the Great Age of American Innovation". I just finished it a couple of days ago and have been telling everyone at work about it. Highly recommended!

Embedded Video

For the last couple of years people have been asking me to do videos on subjects of interest to the embedded world. The occasional requests have grown to a chorus, so, with some trepidation, I've decided to start a video series on my web site. A new one will be added weekly (perhaps with some exceptions for vacation!).

Episode 1 is "Why Did My Board Crash When I Scoped a Node?" Fast digital signals affect all developers, not just hardware people, and "fast" does not necessarily mean "high clock rate." In this video Jack demonstrates how transitioning signals can create havoc for people debugging hardware and firmware, and what strategies one can take to mitigate the issues.

There will always be a link to the latest video here. A complete list, though sparsely populated today, is here.

Quotes and Thoughts

"If you want to set off and go develop some grand new thing, you don't need millions of dollars of capitalization. You need enough pizza and Diet Coke to stick in your refrigerator, a cheap PC to work on and the dedication to go through with it." - John Carmack
Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

The Case of The Termination

In Muse 253 I ran a story I wrote back in 1994 about Jake, a Baltimore private investigator, in The Case of the Crashing 68000. Here's another episode in Jake's life from 1996.

A scruffy pair of wingtips were planted two inches in front of my face. I moaned, rolled over, feeling bits of dried blood flake out of my hair. The sour taste of last night's whisky reminded me that somehow I hadn't made it to that AA meeting.

"Wake up, Jake." One wingtip prodded my nose. "You lousy bum. I oughta have your PI license pulled."

Lenahan stretched out a hand, yanking me roughly to my feet. "Ain't ya got some speeders to bust?" I asked, trying hard to maintain my balance. Taking stock it was pretty clear to me that balance was all I could ask for, my dignity obviously long gone. Feeling my pockets the night slowly came back. Wallet - gone. She was pretty but greedy. .45 - also missing. Hazily an image of trading it and the brass knuckles for a bottle of hooch formed in my still spinning brain.

"Angel's gonna be pissed at ya, Jake. You promised her you'd lay off the sauce." My leggy receptionist was a crusader, out to reform me, not quite understanding how impossible her mission was. I encouraged her, needing her support and willingness to miss paychecks far too often. Lenahan is right - I am a bum.

"Your child support payments are six months behind, Jake. I'm taking you in."

We'd been through much together, Lenahan and I. He spared my dignity by not cuffing me for the ride downtown.

Since they re-instituted debtor's prison back in '18 the character of the jails here in Silicon Valley has changed. Violent criminals are rare, most released to make room for the Valley's hordes of bankrupt electronic executives. I'd moved to Baltimore to get away from hanging judge Tito, who gleefully accepted his low-profile role of incarcerating EEs, but all too often found myself out here on the shores of the Pacific.

The joint's like an old home for me; I see it from the inside more than I'd like to admit. The regulars all waved happily. Even while doing hard time they all scheme new scams. You could see it in the graffiti - the walls were covered with schematics and financing plans. "Jake, ya got any overhead transparency material?" a young offender whispered. "I can trade ya a carton of smokes."

A few smuggled whiteboards were propped in hidden corners, their owners crouched with fistfuls of multicolored markers, glimpsing furtively over their shoulders every few seconds for a sign of the screws.

Evidence of Spike's last test equipment caper were everywhere. Smuggled scopes cast a blue glow over the dirty ceiling. "Psssst - ya got a 20L8?" one gnarled veteran asked in a barely-audible murmur. The poor soul must have been 35 at least. "Heck, who uses them anymore?" I replied, "where've you been, old-timer? It's all FPGAs now." He shuffled back to the bunk, grumbling "they keep stealing my copies of EDN."

I looked up as a trusty idly pushed a cart down the aisle. "Candy. Smokes. Xilinx software" he chanted. Several prisoners greedily grabbed for the latest disks. "Register as John Doe," the pushcart vendor hissed, "you'll get your annual maintenance contract cheaper that way."

Three of us shared a tiny room lined with bars. One cell mate was a small man with thick glasses and tasseled shoes. He seemed out of his depth, a novice unable to cope with the hardened prisoners. "I was a venture capitalist" he sobbed, " and they got me for copying Office onto my computer at home. Now these brutes shove business plans at me all day long. I can't take it anymore. I'll do anything to get out of here."

The other seemed tougher. His pocket protector showed him to be a man of depth, one who was no stranger to the seamy side of the Valley. "You Jake?" he asked, the question framed like a command.

"Yeah."

"I need a little job done. This pencil-necked geek" - pointing to the quivering VC - "promised funding for my new product. We get the prototype working and The Organization will take care of marketing it for us. I got an engineer working on it now, and it's almost ready to go. Whatdaya say we bust outa here and you help me get it working?"

I had to do something before the DTs set in. "Sure" I agreed, and we set to work on the jail's alarm system.

The county-installed system had been improved by countless inmates. Engineers just can't keep their hands off anything, making lousy stuff good and good stuff better. I had enough 10K resistors and clip leads in my pocket, though, that bypassing it wasn't too hard. These folks were not here because of their competence, and the alarm system was now as buggy as the products they designed on the outside.

With the alarm deactivated we slipped down through the laundry room and out through the yard. The guards were gone from their posts, drawn away by a protest over the low-bandwidth Internet connection. Nobody bothered us as we marched out the front gate.

Until, that is, a long white limo screeched to a halt, blocking our view of the street. The door flew open and a meaty hand grabbed me around the throat, drawing the three of us into the car.

"Youse check bounced," Bruno growled, a red flush working its way up from his over-starched shirt collar. "You louse. I oughta off you now."

"Wait" I squeaked. "I'm good for it, Bruno, really I am. My associates here are just about to do a killer application. You can have half of my share."

"70%, and I get two seats on the Board," he demanded.

I remembered why I had moved to Baltimore. The Valley is a tough place, where the numbers do the talking, the IPO is the goal, and a stratospheric price/earnings ratio is everyone's dream.

I needed a drink, bad, but Bruno had his driver set off at once for the lab. Bruno is the Valley's highest paid electronics consultant, and with him in our gang there was no question we'd get the product to market.

We walked down the hallway, technicians scurrying away from Bruno's shotgun eyes and terrified by his reputation. Like an out of control school bus he barreled through every obstruction and banged into the lab. "Where's da system?"

"Ah... here... Bruno, ah, Mr. Gessapetti." The terrified engineer pointed to a bench cluttered with equipment and debris.

"Youse slobs. Hows a artist gonna work with all dis crud around?" Bruno's tree-sized arm swept an Uzi off the bench onto the ground, along with a dozen cans of Jolt and an equal number of Twinkie wrappers. On hitting the floor the weapon fired one burst. Though three people were hit they were only co-op students, so no harm was done. After all, weren't they here to get a taste of what the industry was all about? The corporate downsizings back in '19 had set a new tone in business indifference. I pried the stock options from their hands and hid them in my coat.

Spike raised his ever-present flask to his lips, tilted his head back, and took a long slow swallow. "It's like this, Bruno," he belched, "no matter what we do, we keep blowing up chips. And I mean blowing them up. Danny the Dynamiter saw the Beta 1 unit smoke and grabbed it. He ran outa here hours ago."

I had heard it on the street. Danny claimed to have gone straight, "doing parties" - whatever that meant - but now he was clearly up to no good.

"It's gotta be a static control problem" he went on. "We replaced the carpet with anti-static tile, use ozone generators, and even put our own anti-static workstations together." Waving his hands in the general direction of a line of the assemblers I noticed a dozen Slavic faces, their ankles all cuffed to the benches. "We realized the handcuffs are metallic, and it sure increases productivity while eliminating static."

So this was where the $7/hour engineers were. Now that DOD no longer tolerated underbidding on contracts, these poor souls must have been downsized into the mob. Their mournful chanting of The Song of the Volga Boatmen punctuated distant shouts of "just ship the damn thing."

I made a mental note to talk to Spike about modern management techniques.

Bruno's rumbled "whatcha doin, makin assumptions? You assume it's static? Da last guy dat assumed stuff in my outfit is now pushin up daisies. Youse need data and youse need a open mind. Use yer brain."

A midget head perched atop his enormous body gave lie to the advocates of phrenology, as I knew Bruno is as smart as he is rich. I could also see the wheels turning as he examined the circuit board.

He lightly dragged the back of his hand over the chips. "My fingers don't feel no heat no more," he muttered, "too much soldering. Da back of da hand is much more sensitive to heat. Dis chip is about ta go." A rancid smell announced a stream of smoke from the part's plastic package.

Pouring over the schematics the red started moving up from his collar again. I backed away, secretly wishing I were back in the slammer, and not the possible victim of his rage.

"Youse driving a cable with fast logic," he roared, "what kinda amateurs you got here? I betcha there's no termination on the receiving end!"

I'd seen this performance before. The scope probe was almost lost in his big hands, as he followed the signal as it propagated between boards. With each test he moved the ground clip, a pathetically short segment of wire I knew he used to keep signal integrity.

"Look here. Dere's 2 volts a undershoot at the far end of dis." Spike's protestations that the path was only 10 inches overall did nothing to soothe the enraged Bruno. He angrily threw the probe on the floor, commanding Spike to put a Thevenin termination on the receiving gate. "Yeah, right, static. Static. Static my..." his voice faded as he slumped in a chair.

Bruno had forgotten more about fast logic than I'll ever know. I sat with him, trying to understand. "Jake, ya gotta understand. Them chip vendors is shipping fast parts," he complained. "Da stuff youse buy today is much faster den da stuff we bought back when we was just learning to hot wire cars. Even wid da same part number. Ya gotta think of every signal as a transmission line."

Looking for the bar I spied Danny the Dynamiter standing alone in a corner, a maniacal grin lighting his face, a control unit trailing wires in hand.

"Angel!" I yelled, tackling her and skidding under the table.

A second later the Piñata exploded, showering SOT-23s everywhere.
More on Rate-Monotonic Scheduling

As hoped, readers had some very interesting comments about RMS, which I discussed last issue. Before getting to those comments there is at least one tool that claims to accurately determine timing of code completely by analysis. Absint looks very interesting, and does support Cortex M and other commonly-used embedded parts. I haven't tried it - does anyone have experience with this tool?

Nick P., Security Engineer, wrote:

"WCET can only be determined ex-post facto - after the code is written. One does a careful design, tedious implementation, and only then, late in the game, do we know if our system will be reliable."

It has to be done after the code is written because it analyzes the code. That makes sense. The other part you mentioned is a limitation of the tools, not WCET in general. I think it can be automated at language or tool level as I told you before. So, here's another attempt at a possible solution while maintaining interrupts.

1. The most primitive operations of the machine are timed by something akin to fuzz testing. The timing range is recorded. These include registers, instructions, context switches, stacks, I/O, etc. This portion must be ported to each architecture or device. It can also be done in batch form overnight.

2. A tool processes your code noting the operations, control flow, etc.

3. The tool works bottom up to produce worst case execution profiles based on the operations & timing data of target system.

4. The tool might accomplish this (or speed it up) with techniques like those used to find defects in model checking, abstract interpretation, etc. I mean, the stuff tools like Astree and SPARK prove seems way harder than a timing analysis of straightforward code. I could be wrong.

5. The tool gives you a report showing various WCET results and/or gives you a GUI/shell to ask for data on specific executions. It might even run through many combinations on its own to find a counterexample.

6. The tool might be modified to allow programmers to add annotations to each function that gives a maximum execution time or other policies that the tool checks automatically.

7. Like in some other IDE's, this analysis might be done incrementally and in the background right as you save a module. For example, the 1st processor core is processing what you're doing to preserve flow, while the 2nd core is doing various jobs on the code you write from compilation to regression tests to timing analysis. It can also be a part of build system, but you want quick results so I put it in IDE instead.

So, that's what it would take. I promise you I've seen much more complex functionality in IDE's. I'm guessing the two commercial RMS tools I've heard of don't do it or they're too expensive. If you guys don't have it yet, it just means nobody is taking the time to build the thing. Maybe a bunch of you should get together putting in time here and there to build an OSS (or low priced) tool for this. Then, like with some other embedded tools, each project submits their timing & configuration data to a repository to reduce work for everyone.

Alternative: Put in in the language.

Rather than explain this I'll give you another paper that builds RMS into C++ via metaprogramming.

A language approach can be combine with analysis tools I mentioned if you use an easier-to-analyze language than C++. Especially one designed for verification.

Alternative:

FIX PROBLEM AT ITS SOURCE: INTERRUPTS

There's event-driven architecture and interrupt-driven architecture. They're not the same thing no matter how many people think they are. When I discovered mainframe's Channel I/O [1], I wondered why we live with workstations and servers without it. The work that gets interrupted constantly runs on a dedicated chip with high-speed I/O performance, separate channels for various I/O, etc. It only interrupts main CPU & system software upon events that system designer determines are important. These events are often more high level: the file write is done vs the disk head has moved this far over so many fractions of a second. It's why mainframe's compute processor utilization is often 97-99% despite tons of I/O coming in. And they meet real-time transaction processing requirements as well.

Now, hard real-time embedded systems certainly have cost issues. One can't throw full-on Channel I/O in all their boards. However, you *can* use at least two chips with one focusing on application & other dealing with interrupts and I/O. The simplest method is queuing them in a part of memory (SRAM ideal), while main CPU periodically just pulls them all in at once. It might do the full data over PIO, a pointer to it in memory, or whatever you want to do. It then processes them according to a scheduling policy. It will likely also send a batch of outgoing data during periods it handles I/O. Timing or scheduling can happen at many layers.

I repeatedly used this trick in high assurance security engineering to get untrustworthy stuff, especially device drivers, out of the TCB. A binary blob wants to do unpredictable, unsafe stuff in kernel mode? Onto a super-cheap, memory-restricted coprocessor you go! What about performance and timing you say? Well, I'll leave it to your imagination about how much faster and predictable my system code runs without tens to hundreds of thousands of interrupts from storage and network devices. ;)

Good thing is embedded people are coming around to this. The Sandia Secure Processor (SSP), aka "Score" processor, is a high assurance JVM in silicon [2]. It's aimed to raise both safety & security from instruction set up. Relevant to our discussion is that they use an interrupt-less architecture [3]. Their SEED architecture uses an event-queue model like I suggested and with way less circuitry than my own method. Far as getting things to work well, look at their reported ASIC production results. Amazing. And they have extremely detailed timing data, software simulators, FPGA simulators, and automated + manual tools to help check that all is well each change they make. Could conceivably make WCET analysis easier.

In conclusion, my theory is you embedded guys should just throw out interrupt-driven architecture in exchange for alternatives. My experience shows a coprocessor approach can be pretty cheap. SEED architecture shows it can be simple & amenable to easier timing analysis from hardware up. On high end, Channel I/O architecture shows it can handle massive throughput while meeting timing requirements as well. Using architectures that make the job harder by their very design will... only make the job harder. Make your job easier and Just Say No to interrupt-driven architecture. :)

[1] http://en.wikipedia.org/wiki/Channel_I/O

[2] http://www.cs.utexas.edu/users/jared/ssp-hase-submission.pdf

[3] ftp://ftp.cs.kent.ac.uk/people/staff/phw/.old-1999/tmp2/443-cpa2007-wickstrom.pdf

John Carter

One project started off on the "There is More Than Enough CPU time" trajectory... The product used message passing, between real time absolute priority threads, and run to completion message handlers on ECOS RTOS.

Which eventually failed, and I was given the task of working out why. So I dug out ye olde RMS theory and put the maths into our context and explained the problem to my colleagues. So being able to explain the bug is an important first step, but then what?

"Always on" metadata.

So I redid our messaging framework slightly (completely different under the hood, not a byte changed above). I made each mailbox an in place ring buffer (no mallocs and free, so much faster), and packed a bunch of metadata into the front of every message. (Mail Box X Ring Buffer == RingBox). I packed in a "from" task id, the program counter of posting function, posting time, start handling time, handled time. I was extraordinarily careful and cunning not to screw up the real time behaviour. When each message is finished being handled... the meta data is copied off into a system wide history ring buffer. All this ticks along "always on". "Always On" is _very_ useful. At any stage I can break in with a debugger and slurp all this metadata down and post process.

Analyse and Visualize.

I pour the data into a well normalised http://www.sqlite.org database.

By the way, want nifty free tools?

Sqlite combined with the sqlitemanager Firefox addon is a very easy to use must for anyone with a serious amount of data to query. https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/

(Side note: Why on earth would a web browser have a world class DB query GUI embedded in it? Aha! Firefox uses sqlite for all it's persistence requirements!)

For a visual impression, I query the DB and reformat for input to mscgen, a message sequence chart generator.

http://www.mcternan.me.uk/mscgen/

But seriously, the visual impression is just a learning tool, the real answers come from the SQL "Views" I have created.

These calculate all the parameters of the Rate Monotonic Model, based on real data from the system. Don't let the NSA fool you... give me the Metadata and some good query tools.... and I can and do tell you exactly what is going on! I have lost count of the really really gnarly bugs I have solved with this setup.

Of course, the trick is not to inject the bugs in the first place...

Assert on Designed for Model Parameters.

So I have added a "maximum time in handler" assert to the framework.

Every thread now has an appropriate maximum latency associated with it... and I throw an assert if we hit it. Ok, we have some work to do still, those maximums are still too loose... but we know it is not getting worse, we know where the problems are.

Still, there are some hard problems...

Correlation ID and event cascades.

Under this sort of architecture, an event happens, and triggers a cascade of actions and events to handle it.
Ultimately _every_ cascade of events has, at it's root, some hardware. Thus I have hooked into _every_ hardware driver we have, something that increments a system wide counter.

Every driver that sends a message tags the message with this "correlation id", this is copied if a message handler sends a message. ie. You can use this id to "colour" every cascade of events and track back to the hardware event that triggered the cascade. I have SQL views that will tell you which cascade is the most expensive.

Busy Bees

Sometimes I point at the problem and they say, "But we need the product to do that continuously, as fast as possible!"

These are Busy Bees. Busy bees are sequences of messages that ultimately loop around continuously, and you want them to run as fast as possible. Inevitably something somewhere is going to be starved of CPU time. So I have created a "Busy Bee Breakpoint". You find a point in the busy bee loop that isn't real time sensitive, and you bounce a message via the lowest priority task. The busy bee will continue buzzing away as fast as possible... once the rest of the system has caught up.
Jobs!

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words.

 
Joke For The Week

Note: These jokes are archived at www.ganssle.com/jokes.htm.

Thomas Osbrink was asked to wait 4 years before re-entering his password:

Joke
Advertise With Us

Advertise in The Embedded Muse! Over 23,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.