You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Comments
• Entropy Reduction
• Jobs!
• Joke for the Week
• Advertise with us
• About The Embedded Muse

It happens every time I do a public seminar. At the last minute several participants cancel as a project is in trouble. The engineers are needed to put out the fire. They can't take the time to learn how to avoid the conflagration in the first place.

I am doing public versions of my Better Firmware Faster seminar in Baltimore on Nov. 7, Santa Clara Nov.14, and Germany Nov. 28. No doubt there will be some cancellations, and I salute the engineers who have to go off and save projects. But, we do need to learn better ways to build these systems. As the saying goes, when the boat is sinking, stop bailing and plug the leak!

When I started in the embedded industry there was just one 8 bit micro - the 8008. It had a 16 KB address space, but memory was so expensive programs were on the order of 4-6 KB. The worst ad hoc methods worked fine for those small programs. Now even in the MCU world half a MB of flash isn't uncommon on a 32 bit CPU with fiendishly complex I/O. The firmware is composed of OSes, communications stacks, file systems and all sorts of application code that interacts in complex ways. Ad hoc no longer works. The result is, well, in fact studies show embedded code is much better than any in the rest of the computer industry. Did you know that, on average, 95% of all of the bugs in firmware are removed pre-shipment, compared to 85% for the rest of the computer business? But 95% means too many bugs are getting out the door. We've got to do better.

Did you know the average firmware project spends half the schedule in defect removal? That's far too high.

The Toyota case should have been a sobering moment for the industry. You'd think a $1.2 billion fine would have been a wake-up call for senior management. Alas, firmware is not on their radar, so it's up to those of us in the trenches to improve our skills to avoid these sorts of appalling events.

Join me to learn how to greatly reduce bug rates while slashing the schedule! Over 5000 engineers have taken this class on six continents.

This is a very-fast-paced full day seminar. It's fun, it's informative, and it only covers practical ideas you can use today. The Baltimore and Santa Clara seminars are quite close to airports with Southwest Airlines hubs.

The seminar covers these subjects:

Languages: C, C++ or Java? What about code reuse - is it a myth? How can you benefit? How to control stacks and heaps.

Structuring Embedded Systems: Manage features... or miss the schedule! Using multiple CPUs. Five design schemes for faster development.

Overcoming Deadline Madness: Negotiate realistic deadlines... or deliver late. Scheduling, the science versus the art. Overcoming the biggest productivity busters.

Stamp Out Bugs: How to manage bugs to get good code fast. Quick code inspections that keep the schedule on-track. Cool ways to find hardware/software glitches.

Managing Real-Time Code: How to design predictable real-time code. Managing reentrancy. Troubleshooting and eliminating erratic crashes. Building better interrupt handlers.

Interfacing to Hardware: Understanding high-speed signal problems. Building peripheral drivers faster. Inexpensive performance analysis.

How to Learn from Failures... and Successes: Embedded disasters, and what we must learn. Characteristics of a healthy firmware team. A seven step plan to firmware success.

There's more information on all three public seminars here. Or, I can deliver the seminar on-site, at your facility, just to your engineers.

I have always wished that my computer would be as true. I no longer know how to use my telephone. - Bjarne Stroustrup

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Adam Taylor has been doing a weekly blog about Xilinix's MicroZed board, which is an evaluation board for the Zynq FPGA. The Zynq has two Cortex A9 cores in addition to the FPGA fabric. He covers the basics of starting out all the way up to multiprocessing and the addition of several different operating systems. He has recently completed a year of posting; 52 episodes in all, which are here. I have a Microzed and was planning a short review, but Xilinx convinced me to download their toolchain and really dig into it. That will take some time so won't happen till late in the year.

Brian Voskerijian sent this link to an interesting article about some of C's quirks.

I was chatting informally with a group of developers recently when one used a phrase guaranteed to set me off.

"I write self-commenting code," he revealed proudly.

Uh huh. I have yet to see a program of any useful size that, stripped of comments, is self-documenting. C is not a language like English or Swedish where there's so much information conveyed that even in a noisy room, where one might catch only 70% of the words, the meaning still comes across. Computer languages are inherently dense and precise: miss a single character and the program won't run correctly. Mix up "Identifier" and "identifier" and, if you're lucky, the compiler will complain.

Usually these folks will go on at length about their use of long variable names. Hey, I'm all in favor of making variables as long as they need to be to clearly express an idea. But length, in this case, isn't always an asset. I find it awfully hard to read something like:

for(next_output_buffer_sequence!=first_output_buffer_sequence; 
  !complete_message_assembled_by_host_process; ++final_result_queue_pointer);

The C constructs (operators et al) are lost in the morass of names. And reading a single statement split across many lines confuses the eyes.

Then there's the 31 character uniqueness limit so it's dangerous to get enamored with exceedingly long names.

Some developers subscribe to the "comment nearly every line" school of thought. Their code looks like:

for (i=0; i<max; ++i){   // Start initialize loop
   ++Array_Pointer;      // point to next element in Array
   *Array_Pointer=0;     // Set Array element to zero
   }                     // end for

Much better is to eliminate those annoying and not particular i
nformative comments and prefix the entire snippet with:

   //  Array is a sparse matrix; empty elements are denoted by zero so
   // here we initialize all elements to "empty" (a zero).

The second style conveys the sense of the code while the first gives plenty of detail and no context.

Others write the code first and add the comments later. They don't want to "waste time" with documentation while endlessly fiddling with a routine to get it to work. But the comments should be the function's design. If the developer doesn't know enough about the design before cranking code, just how do he start pounding C into the editor? Is it a random walk? "Uh, hmm, I dunno, let's try:"

void main(void){

"Oh boy, now what? How about maybe initializing something... or should we set up a queue?"

The idea must be that if they type enough C a function, a structure and a clear idea will emerge. That might indeed happen... eventually. But not efficiently.

There's a spec somewhere -- perhaps only in the developer's head -- which describes in English what a function should do in a human-friendly manner. The code is a translation of that spec to cryptic and unforgiving computerese. So I figure the way to write a function is to create all of the comments first. The header, and even all of the individual little snippets of English spread throughout the code. Then the function's design is done.

After that, anyone can fill in the code.

Old fashioned air-cooled Volkswagons required an oil change and valve adjustment every three thousand miles. I change my sailboat's diesel engine's oil and all of the oil and fuel filters every 100 hours of operating time. Every season -- if I think of it -- I change the oil in that lawnmower I loathe so much.

Jiffy Lube has built an international company around providing this sort of preventative maintenance for cars. No one questions the need to regularly invest a bit of time and money to keep engines running well.

It's a shame we're not so farsighted in the firmware business.

Code is like the oil in an engine. When it gets dirty -- nasty, convoluted, patched into an unmaintainable mess - it gums up the application. Eventually, things get so bad the system seizes -- it can no longer be improved since the cost of dealing with the chaos is so high.

Many agile methods push continuous refactoring, or rewriting crummy code. Some suggest we rewrite all code that can be improved. Like world peace, it's a great concept that isn't really possible in the grimy trenches of getting a product out the door. Perfection, as desirable as it is, will never override all other considerations.

The second law of thermodynamics says that disorder increases in closed systems. Entropy increases. Programs are no more exempt from this depressing truth than the expanding universe. Successive maintenance cycles increase the software's fragility, making each additional change that much more difficult.

Software is like fish. It rots. Over time, as hastily-written patch after panicked feature change accumulate in the code, the quality of the code erodes. Maintenance costs increase.

As Ron Jeffries has pointed out, maintenance without refactoring increases the code's entropy by adding a "mess" factor (m) to each release. That is, we're practicing really great software engineering all throughout a maintenance upgrade... and then a bit of business reality means we slam in the last few changes as a spaghetti mess.

The cost to produce each release looks something like: (1+m)(1+m)(1+m)...., or (1+m)n, where n is the number of releases. Maintenance costs grow exponentially as we grapple with more and more hacks and sloppy shortcuts. This explains that bit of programmer wisdom that infuriates management: "the program is too much of a mess to maintain".

But many advocate starting release N+1 by first refactoring the mess left behind in version N's mad scramble to ship. Refactoring incurs its own cost, r. But it eliminates the mess factor, so releases cost 1+r+r+r..., which is linear.

Luke Hohmann calls this "post release entropy reduction." It's critical we pay off the technical debt incurred in being abusive to the software. Maintenance is more than cramming in new features; it's also reducing accrued entropy.

It's sort of like changing the oil on a regular basis to keep things humming smoothing along.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words.

Note: These jokes are archived at www.ganssle.com/jokes.htm.

LR sent this: Although the speed of Java has increased dramatically in recent years, claims that it can be faster than C are not true, because as we all know, Einstein proved that nothing is faster than C.

Advertise in The Embedded Muse! Over 23,000 embedded developers get this twice-monthly publication. .

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.