You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Apollo Software - Pretty Darn Good
• Building Maintainable Software
• Jobs!
• Joke for the Week
• Advertise with us
• About The Embedded Muse

The quality movement’s central observation is that building in quality inevitably results in lower costs. That’s the focus on my Better Firmware Faster course. It is possible to accurately schedule a project, meet the deadline, and drastically reduce bugs. Bring this class to your facility.

In Quality Software Management Gerald Weinberg claims that multitasking between three projects incurs a 40% context switching overhead. That means your 40 hour week is really only 24 hours of useful work. To look at this another way, over 800 hours a year – 20 work weeks – would go to context switching.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

I reviewed Rigol's new DSA815-TG spectrum analyzer on embedded.com last week. It's a nice unit with a great price.

Mat Bennion sent:

Jacques wrote:

Doug LaRue had some power supply issues:

Time Benson suggests UMLet:

A number of people objected to an offhand comment by one Muse reader that disparaged the code in the Apollo Guidance Computer last issue. They're right - the AGC and its code were remarkable for the day. Can you imagine designing a computer with just 2300 dual NOR gates for landing on the moon? Apollo was a brilliant success, one that touched my family. For instance, my dad was surprised to find that it was illegal to shovel Jones Beach sand into the back of his station wagon.

It was 1958. Two successful and surprising Sputnik launches had energized the political class and spawned engineering efforts to match or beat the Soviet's success. All the US had managed to loft was Explorer 1, which, at 31 pounds, was a mere 3% of the mass of Sputnik 2. But even in those very early days of spaceflight, when getting anything into even a near-Earth orbit seemed impossibly difficult, work was being done on a manned lunar landing. At Grumman on Long Island my dad was doing pre-proposal landing studies. They thought sand could simulate the lunar surface. The cop somehow didn't believe his wild story about traveling to the moon, but simply issued a warning and moved on, no doubt shaking his head in disbelief.

The word "lunatic" comes to mind.

But just over a decade later two astronauts managed to land and return safely. In another couple of years the country was bored by the concept; I remember watching a later mission blasting off in a quarter of the TV screen while a football game filled the rest.

How did Kennedy's mandate succeed, especially given the primitive technology of the era? How did armies of mostly young engineers invent the spacecraft, the web of ground support infrastructure, and the designs of the missions themselves in such a short time?

The rich history of space exploration has been well-told in many books. My favorites include Carrying the Fire: An Astronaut's Journeys, by Michael Collins, the lonely pilot who stayed in orbit during Apollo 11's descent. Of all of the astronaut autobiographies this is the most literate and finely crafted.

Then there's Apollo 13 by Jeffrey Kluger and James Lovell. Lovell, of course, was the commander of that ill-fated mission. Though sometimes seen as a disaster, 13 strikes me as a Macgyver-esque tale of success. After big explosion 200,000 miles from Earth, in the depths of space, their consumables running out, the crew and engineers on the ground cobbled together fixes that brought the mission home. That the fragile craft could sustain such major damage is a testimony to everyone involved in the program.

Tom Wolfe's The Right Stuff is the story of the astronauts themselves. Though innumerable books laud their bravery Wolfe captures the existential nature of the test pilot catapulted into a situation where mistakes just aren't an option. Like the others I've mentioned, it's highly readable and engrossing.

But these books, like most of the others on the market, are largely about the pilots who rode into space. They're shy on technical details. Just how did they navigate to the moon, anyway? Whizzing along at 2000 miles/hour it moves about a quarter pi radians around the Earth in the three days of flight. How did managers grow the program from a dream to hundreds of thousands of workers and $25 billion in so few years?

Enter Apollo: The Race to the Moon by Charles Murray and Catherine Bly Cox. This page-turner (at least for geeks) juxtaposes a picture of a Mercury capsule being haphazardly trucked to the pad, swaddled in mattresses, with the monstrous crawler-transporter ponderously moving a Saturn V. The book is out of print but used copies are available.

One might think that after 45 years this subject would be exhausted, but Digital Apollo by David Mindell falls into the ranks of the best Apollo books for geeks.

It's not unusual for a space book to reach back to the story of the X-15 or even Chuck Yeager's X-1 record-breaking flight, but in Digital Apollo Mindell starts with the story of the Wright Brothers. Yes, they are generally credited with inventing the airplane, but he focuses only on their experiments with wing warping and other forms of control. For the Wrights's pioneering work succeeded largely because they invented ways to steer a flying machine in three axis. And control, particularly as manifested in the Apollo Guidance Computer, is one of the two overarching themes of the book.

Digital Apollo starts with an initially somewhat boring investigation into test pilot psychology. Those flyers formed the cadre from which most of the astronauts were selected. Why should a modern reader care about some jet jockies' inflated self-esteem? Mindell skillfully weaves that discussion into the nature of aircraft stability, instrumentation, and then into his second major theme: Who should control the vehicle? The pilot, or a computer?

He avoids the obvious question of why, if the automata are so good, we bother to include a person in the spaceship. That essentially political and philosophical debate belongs elsewhere. But it certainly goes to the nature of being human and what exploration means in the virtual age. Mindell does end with a captivating description of the 2004 meeting of the Explorer's Club where famous folk like Aldrin, Piccard (not the one of Star Trek fame) and Hillary (not the one in the news) listened to Pathfinder's Stephen Squyres give a talk about how he and his team explored Mars while they were comfortably seated in an air-conditioned lab.

At the very first meeting of the Society of Experimental Test Pilots in 1957 some attendees expressed fear that they would be automated out of a job. Throughout the book it becomes clear that much of the engineering of the various spacecraft was dominated by the tension between traditional rocketeers, who felt everything should be automated, and the pilots. Some Mercury astronauts wanted to fly the rocket right off the pad.

If control is just keeping a needle centered on a gauge why stick a heavy human in the control loop? Those with the Right Stuff want to be in control, to take heroic action, and those desires impacted the space program in the 60s and still do today.

The USSR believed in automation and their authoritarian culture brooked no interference from mere cosmonauts. Gagarin, the first man in space, was blocked from his spacecraft's manual controls by a combination lock whose code was in a sealed envelope!

Despite Mercury being the smallest and most primitive of the Apollo triptych, pilots of those first six missions had the least control of all. All important operations were automatically sequenced. System failures forced Scott Carpenter to assume manual control. He saved the mission, but various delays pushed Aurora 7 250 miles off course. Some saw this as a success for those advocating the need for the pilot in the loop; others, loudly, blasted him for poor performance.

The astronaut office rebelled at Mercury's spam-in-a-can design. On Gemini each mission's dynamic duo still had only systems management responsibility on launch, but were an integral part of the control loop in orbit. Armstrong and Scott saved their Gemini by reading a magnetic tape with reentry procedures into the computer.

The hero had been redefined as a boot loader.

About a hundred fascinating pages into the book Mindell begins to look at Apollo, and specifically the Apollo Guidance Computer (AGC), which was, other than software, virtually the same as the LM Guidance Computer. Recognizing the difficulty of controlling a spacecraft from the Earth to Lunar orbit, and thence down to the surface, NASA let the very first Apollo contract not to the spacecraft vendor, but to MIT's Instrumentation Lab for the AGC.

Oddly, everyone forgot about the code. The statement of work for the AGC said this about software: "The onboard guidance computer must be programmed to control the other guidance subsystems and to implement the various guidance schemes." That's it. Lousy requirements are not a new problem.

Another phenomenon that's not new: once management realized that some sort of software was needed, they constructed a procedure called a GSOP to specify each particular mission's needs. But in practice GSOPs were constructed from the code after it was written, rather than before.

If Doxygen had been around they could have saved a lot of time.

Flash memory hadn't been invented. Nor had EPROM, EEPROM or any of the other parts we use to store code. For reasons not described in the book (primarily size) traditional core memory wasn't an option, so the AGC stored its programs in core rope. Programs were "manufactured" -- the team had to submit completed tapes three months before launch so women could weave the programs into the core ropes that were then potted. Once built it was extremely difficult to change even a single bit.

The code was written in an interpreted language called BASIC, though it bore no similarity to the BASIC later produced by Dartmouth. Listings of the AGC code show that even in the 60s software engineers used random names for labels, like KOOLADE and SMOOCH, the latter suggesting perhaps too many hours at the office.

There was never enough memory (32K for code). Then, like now, developers spent an inordinate amount of time optimizing and rewriting code to fit into the limited space. Consider that a tiny 32K program controlled the attitude indicator, all thrusting and guidance, a user interface, and much more.

And what was the user interface? Parts of the UI included windows (not on a desktop, but those glass things people peer though) that were etched with navigational info. The computer prompted the astronaut with angle data; he looked out through the markings and designated new targeting parameters to the computer. Did the computer have an UI... or did the pilot? Who was driving what? The pilot was just another component in the system.

Mindell paints the moon flights as a background patina to the guidance theme. An entire chapter is devoted to Apollo 11, but other than describing how the astronauts used the stars to align the inertial reference platform the author ignores the flight to and from the moon. The chapter covers the landing sequence in detail -- some might say excruciating detail but I was riveted. Most remember the 1201 and 1202 alarms that caused great concern. It turns out that Aldrin wanted to leave the rendezvous radar (which was only needed to seek out the command module) turned on in case an emergency abort was needed. Program management signed off on this change, but no one told the software engineers. Data from the radar demanded computer attention that was in short supply, so it flashed the 1201 and 1202 codes indicating "I'm getting overwhelmed!" The AGC rejected low priority tasks but ran the landing activities correctly. That's pretty slick software engineering.

A single chapter covers the descent and landing phases of the next five missions (Apollo 13 is left out as it never entered lunar orbit). Many are familiar with the abort switch failure, later traced to a blob of solder, that could have ended 14's mission. That's described brilliantly, but the software patch is glossed over in a short barrage of words that left me feeling the author really didn't understand the fix.

This book is short on deep technical details about the AGC. For more on that see Journey to the Moon: The History of the Apollo Guidance Computer by Eldon C. Hall. But it provides a fascinating look at the human side of engineering, and how pilot demands (think "customer demands"), both reasonable and not, affect how we build systems. Those inputs, Mindell thinks, largely shaped the design of the Space Shuttle. He quotes astronaut Walter Cunningham about splashing down in the sea in Apollo: "[the crew were recovered] by helicopter like a bag of cats saved from a watery grave." And Cunningham's take on Shuttle: "[it] makes a smooth landing at the destination airport and the flight crew steps down from the spacecraft in front of a waiting throng in a dignified and properly heroic manner."

The AGC and its software landed twelve men on the moon. And it brought them all home. Pretty amazing, and way cool.

Charles manning recommended the book Building Maintainable Software by Joost Visser. Who could resist a title like that? And with Amazon Prime and One-Click ordering, well, the book was here a day or two after Charles' email came in.

Each chapter starts with a quote; the very first is from "any programmer," which reads "Who wrote this piece of code?? I can't work like this!!"
I think Mr. Visser misquoted Mr. Any Programmer. Much more likely is "This code is complete crap!"

The book has three underlying principles:
1. Maintainability benefits most from simple guidelines
2. Maintainability is not an afterthought, but should be addressed from the start of the project.
3. Some violations are worse than others.

The second is especially important and mirrors a central observation from the quality movement: if your focus is on fixing bugs you'll never get quality code. It, like maintainability, has to be designed in from the beginning.

It then lays out ten rules for writing maintainable code, devoting a chapter to each. I promise you you'll disagree with some, for instance the very first rule which limits methods and functions to 15 lines of code.

Each chapter justifies the rule, and provides answers to common objections to each rule. The book also gives guidance for refactoring code to ensure the rule is met, though I was a little confused by that as underlying principle 2 mandates getting it all right from the outset. Still, the advice is useful for working on legacy code.

I'm a metrics guy, and the book does a great job of providing metrics for evaluating code to a "four star" basis. I did feel some of the numbers are arbitrary, though, and don't understand what comprises, say, a three, two or one star product. It was surprising to see that for a four-star system only 6.6% of modules are allowed a fan-in of 51 or more. 51! Gads zooks, that would be a nightmarish ball of mud.

Mr. Visser is adamant that comments should be correct, but also implies that comments in general should be avoided as they tend to get inaccurate as the code changes. I (strongly) disagree and feel that as professionals it's our responsibility to keep them current.

Building Maintainable Software is a short work you'll devour in an hour or two. There's lots of good advice, and while you may disagree with the author about some of the rules, his arguments will at least make you think. And we need to think deeply about how we build systems today.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Note: These jokes are archived at www.ganssle.com/jokes.htm.

Morten Kristiansen sent this:

Once a biologist, an engineer and a mathematician observed a house for a very long time. Two people enter the house. A little later, three people exit. "Ah!", said the biologist, "Reproduction, they multiplied". "Nah!", said the engineer, "Measurement error. We counted them wrong". The mathematician said nothing. A little later one person entered the house. "Uuuh!", said the mathematician, "Now the house is empty!"

Advertise in The Embedded Muse! Over 25,000 embedded developers get this twice-monthly publication. .

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.