You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Defects Per KNCSL
• On Superprogrammers
• On Requirements
• Failure of the Week
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Architecture wins over technology - Morris-Ferguson law

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Uh oh. Do you sell to the US government? Since they buy pretty much everything, pretty much everyone does. A new executive order re security will make our lives much, much harder. Though the details are still being fleshed out, a pretty good overview here will raise your blood pressure. The company hosting that article has a product they claim will find (all? most? a few?) security vulnerabilities, though the description is so vague I wonder about its efficacy.

Want to become an embedded developer? Dick Selwood sent a link to a site that will train you for just $6! However, if I'm reading this right, you'll have to invest a staggering 7 hours of your time.

Paul Carpenter sent a link to an article about designing products meant for household use. Increasingly regulators are holding these to the IEC 60335 safety standard which has strict requirements for isolation voltages, creepage and clearance distances, and leakage currents in AC-DC power supplies. Meeting these requirements is not easy.

Peter Gliwa is kindly offering a copy of his book Embedded Software Timing for December's giveaway. I reviewed it here.

Enter via this link.

One of my beefs with Test-Driven Development is that, in my experience, most groups practicing TDD don't use any other process to eliminate defects. But testing will never find all of the problems. Careful software engineering is all about using layers of practices to produce great code. Think of each layer being a filter. None is 100% effective, but taken together they will find the vast majority of problems.

A presentation called "Flight Software Complexity Briefing" from NASA (which I can't find online) has a lot of interesting data, including some eye-opening stats on test. According to the document "most estimates put the number of residual defects for a good software process at 1 to 10 per KNCSL [kilo non-comment source line]". That squares with Capers Jones' data that only 7% of companies ship products with under about 10 defects per KNCSL. His data also shows a paltry 0.4% of us do better than 0.3 defects per KNCSL. That is, the very best organizations ship something like 300 bugs per mega-line of code. (One takeaway from that startling result is that for high-reliability code you better include mitigation strategies to deal with the fact that things will fail).

The NASA brief goes on:

• "A rule of thumb is to assume that the severity ratios drop off by powers of ten: if we use 3 severity categories with 3 being least and 1 most damaging, then 90% of the residual defects will be category 3, 9% category 2, and 1% category 1 (potentially fatal).

• "A mission with 1 Million lines of flight code, with a low residual defect ratio of 1 per KNCSL, then translates into 900 benign defects, 90 medium, and 9 potentially fatal residual software defects (i.e., these are defects that will happen, not those that could happen)."

I'm not sure I agree with the statement "these are defects that will happen, not those that could happen," as some latent defects may never manifest themselves. But it is important to realize that even outfits with way-above-average practices will be shipping code that could be devastatingly bad.

Jones' data shows that the top 20% of great software engineering outfits ship code with as many as 13 bugs/KNCSL, which, using the NASA model, means they ship with over 100 potentially fatal defects in a one-million-line program. Or, that's one guaranteed potentially fatal defect in a little 10 KNCSL product.

As a point of interest, the Space Shuttle's code was probably the best ever written. They did a bit better than 0.1 defect per KNCSL. (See They Write the Right Stuff for info on how the Shuttle developers achieved this).

This chart from the NASA presentation is a bit superficial, but it gives you an idea of the processes they recommend:

Obviously, everyone reading the Muse is a superprogrammer able to crank tall programs in a single bound. But what about other folks on the team?

Turns out, we have some pretty good data about how effective different people are. Consider some from Capers Jones:

Superprogrammers are about six times more productive than the worst folks... on small projects.

Interestingly, NASA's data is not far off, though it does not provide guidance on project sizes:

From NASA’s Manager’s Handbook for Software Development. "Application experience" is defined as prior work on similar projects.

NASA advises using this data in estimating schedules.

The previous picture is pretty illuminating. Everyone is equally awful on big projects. That ties in with Fred Brooks' aphorism Adding people to a late project makes it later. As it does with DeMarco and Lister's observation that big projects are schedule killers because with so many people folks spend all of their time communicating, and not getting much useful work done.

A small project means a tiny team, perhaps one or two people. As things scale the group swells. Pretty soon their days are consumed with reports, emails, meetings, and the like. Occasionally, one gets to squeak out a line of code while waiting for the next meeting to start. The best and brightest handle a meeting at the same speed as a dullard.

Takeaways: segment projects as much as possible into small chunks. Put your best people on the pygmy portions. Protect the superprogrammers from the meeting/email/reporting insanity.

In my half-century in the embedded space requirements gathering has morphed:

• 1970s: We're EEs. Micros are so new. We never heard of the word "requirements" so will wing it.
• 1980s: We're software people. Eliciting requirements is simply part of our job.
• Today: We're software people. We don't talk to the business folks. They don't know much anyway, and it's hard to figure out what people want, so we'll start building something and see how people like it.

It's true that requirements change. But most often not to the extent experienced by people doing, say, web design. Sure, there are tweaks and additional feature requests. And, yes, rarely a revolutionary twist requires a major redesign. But more often the requirements changes we moan about are a product of initial poor requirements gathering.

There’s a meta-meme at work, one that’s ever-more prevalent in today’s vicious political arena. As an engineer I’m constantly appalled to listen to some pundit or soon-to-be convicted politician expound on how we should fix X, where X is really a symptom of some problem. Focusing on symptoms is like bailing a leaking boat. Better: plug the hole first.

And that’s the issue here. Do a poor job at eliciting requirements and the symptom will be a never-ending scramble to patch up the project. Band-aids instead of avoiding the injury in the first place.

In the embedded world, poor requirements gathering is endemic. We jump in and start writing code and designing circuits far too early. Some agilists commend this practice; I think it’s a mistake.

In many cases we blame outside forces. The boss wants us to start today. Marketing keeps making changes (thoughts on that here). There are truly forces at work that may be insurmountable, but all-too-often we’re culprits as well, either in our zeal to get started or our unwillingness to educate the other stakeholders.

Shockingly, many of us know little about requirements gathering. Have you read a single book on the subject? Few of us have. (My favorite is Karl Wiegers’ Software Requirements). I scanned the course catalogs of several universities and found not a single CS or EE course title or description that contains the word “requirements.”

Yes, of course requirements change, and we need strategies for coping. But we have a responsibility to do a great job on eliciting them at the outset to ensure the project doesn't collapse.

I’m reminded of an old parable: In ancient China there was a family of healers, one of whom was known throughout the land and employed as a physician to a great lord. The physician was asked which of his family was the most skillful healer. He replied, "I tend to the sick and dying with drastic and dramatic treatments, and on occasion someone is cured and my name gets out among the lords.

"My elder brother cures sickness when it just begins to take root, and his skills are known among the local peasants and neighbors.

"My eldest brother is able to sense the spirit of sickness and eradicate it before it takes form. His name is unknown outside our home."

There is no glory in getting the requirements right at the outset, but it’s the essence of great engineering.

Steve Wheeler sent this gem:

And this is from Neil Peers - it's the "clocking in" page at his company. They sure hire young!

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.

SEGGER has a new version of the RTOS called "embOS Ultra". Interestingly, it does not rely on a periodic interrupt. Traditionally RTOSes have a timer tick that interrupts the MCU every ms or so; each interrupt may cause task rescheduling but does consume CPU resources, whether needed or not. The Ultra version instead relies on a timer that includes a compare register to cause interrupts only at precise times. Need a task to run every 450 ns? Ultra supports that. This is both more precise in that events can be scheduled with tremendous (down to the cycle level) granularity, and it can free CPU cycles as the RTOS is idle unless something needs to happen. Figure on power reduction, too, for those ultra-low power applications as the MCU does not have to wake up for a periodic interrupt.

FreeRTOS also has a tickless option to reduce power consumption.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors. Full disclosure: SEGGER does advertise in the Muse.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

A school teacher was arrested at John F. Kennedy International Airport as he attempted to board a flight while in possession of a ruler, a protractor, a compass, an ancient wooden device called a “slide-rule” as well as a device called an “abacus” that he claimed was a calculator. At a morning press conference, the Attorney General said he believes the man is a member of the notorious Al-Gebra movement. He did not identify the man, who has been charged by the FBI with carrying weapons of maths instruction.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.