You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com.

• Editor's Notes
• Embedded Video
• Quotes and Thoughts
• Tools and Tips
• On Margin
• Team Sizes
• Jobs!
• Joke for the Week
• Advertise with us
• About The Embedded Muse

Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it’s not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm.

Gin, the cops and core memory. Who would have thought they'd all go together?

"We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next 10. Don't let yourself be lulled into inaction." Bill Gates

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

William Leara has an excellent write-up about a GUI for the Uncrustify code beautifier here.

The video made my jaw drop. In 2005 Flight 292's nose gear was cocked sideways, twisted 90 degrees from its normal position. As the pilot began his approach I wanted to turn away but was glued to the screen in fascinated horror. The wheel touched down, smoked, burst into flame and the tire tore away, nothing but metal grinding along the runway.

Astonishingly, the strut held fast.

Those seconds summed up a lot of the nature of engineering. The strut held fast, for loads that far exceed anything the plane experiences in normal operation. Engineers designed enough margin into the system to handle almost unimaginable and unanticipated forces.

It also shows the human side of the mechanical world. This failure had apparently been experienced before by other Airbus 320s. Something went wrong with the system used by the air industry to eliminate known defects.

I'm struck by the difference between failures in mechanical systems and those in computer programs. Software is topsy-turvy. Mechanical engineers can beef up a strut to add margin, to handle unexpected loads. EEs specify components heftier than needed and wires that can take more current than anticipated. They handle surges with fuses and weak links.

In software if just one bit out of hundreds of millions is wrong the application completely crashes. Margin is difficult, perhaps impossible, to add. Exception handlers can serve as analogs to fuses, but they're notoriously hard to test and generally have a bug rate far higher than that of the application.

Worse, we write code with the assumption that everything will work and there won't be any unexpected inputs. So buffer overflows are rampant. This complacent attitude isn't exclusive to desktop developers; after a software error destroyed Ariane 5 the review board cited a culture that assumed software can't fail. If it works in test it will work forever.

A plane, bridge and dare I say levee must have a reliability vanishingly close to 100%. So mechanical engineers design a structure that takes 110% or 150% of expected loads.

Many software apps require just as much reliability. But we can't add margin, so must build code that's 99.999% correct or better.

Yet humans aren't good at perfection. In school a 90% is an "A". If our code earned an "A," a million line-of-code program would have 100,000 errors.

Software is inherently fragile. We can, and must, add great exception handlers and use the very best methods to produce correct code. But until we find a way to make code that is more robust than the environment it's in, the elusive goal of perfection is our only hope.

What do you think? How can we add design margin to code?

When the IBM 360 tools project started running late managers hired more programmers. And then they iterated this process, eventually boosting team size from the projected 150 to over a thousand developers. They found that each new batch of hiring slowed the project dramatically. Fred Brooks recounts in his seminal must-read "The Mythical Man-Month" that the effort eventually ran years late and dramatically over-budget.

As a result of this debacle he formulated the thesis now known as Brooks Law: adding people to a late project makes it later.

Barry Boehm's Constructive Cost Model (COCOMO), used for estimating software schedules, mirrors this finding. As projects grow in size individual productivity plummets. There's an exponential relationship between the schedule and the size of the project as measured in thousands of lines of code. Typically COCOMO predicts that as a program swells to the million line of code mark programmers will each crank just 17 lines of code per month.

The reason is communication, of course. A small project done by one lone developer has zero comm channels... it's all in Joe's head. Two folks have a single link. The number of channels, if N is the number of developers, is N(N-1)/2, practically an N² rate. Pretty soon, with enough people, everyone is so busy attending meetings, responding to email, writing reports and it's rare they have time to leak out a line or two of code.

Windows Vista was embarrassingly late. Could Microsoft's army of developers been drowning in too much communication?

Of course it's impossible to deliver 50 MLOC with only a dozen people on the team. But it is possible -- perhaps mandatory -- to partition projects into small bits and pieces, each logically distinct, each of which can be implemented by a handful of programmers.

What's the situation in the embedded world? UBM's 2013 survey of the embedded space pegged average teams sizes at 14.6 people. It's not uncommon to have a million lines of code anymore, which is a huge effort likely to consume the efforts of hundreds of developers.

Yet teams of that size are fairly rare. A lot of these giant projects are organic, having evolved over the years from smaller apps as features and functionality get added from one release to the next. Sometimes a newly acquired company has a big code base that gets integrated into the buyer's products. I've run into a couple of groups managing humungous chunks of code with only a handful of developers. In another notable exception there are 600 developers maintaining one legacy code base.

Today's business mantra dictates doing more with less. But that doesn't square with skyrocketing code sizes.

How big are your projects... and how many developers do they use? Are there enough... or too many?

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words.

Note: These jokes are archived at www.ganssle.com/jokes.htm.

In response to last issue's take on engineering humility, Charlie Moher sent this true story:

NEVER LAUGH AT A CUSTOMER' S ABSURD MAINTENANCE SUPPORT CALLS !!!

At a Canadian telecom company we had a customer call in and report that their telecom equipment never worked unless someone flushed a toilet and then it was only for a few minutes afterwards. Fortunately, I resisted the temptation of howling in laughter or casting doubt on my customer's report. We dispatched a tech to the customer site and just like they said, none of the telecom equipment worked until a toilet was flushed. Further investigation revealed that the problem started after a plumber had done some work. He replaced a chunk of copper, sewer pipe with PVC. That copper pipe was their ground connection. The only time they got a ground was when there was a conductive water bridge in the PVC pipe.

Advertise in The Embedded Muse! Over 23,000 embedded developers get this twice-monthly publication. .

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.