By Jack Ganssle

Engineering Apollo

Published in Embedded Systems Design, September, 2008

My dad was surprised to find that it was illegal to shovel Jones Beach sand into the back of his station wagon. Was he going to jail?

It was 1958. Two successful and surprising Sputnik launches had energized the political class and spawned engineering efforts to match or beat the Soviet's success. All the US had managed to loft was Explorer 1, which, at 31 pounds, was a mere 3% of the mass of Sputnik 2. But even in those very early days of spaceflight, when getting anything into even a near-Earth orbit seemed impossibly difficult, work was being done on a manned lunar landing. At Grumman on Long Island my dad was doing pre-proposal landing studies. They thought sand could simulate the lunar surface. The cop somehow didn't believe his wild story about traveling to the moon, but simply issued a warning and moved on, no doubt shaking his head in disbelief.

The word "lunatic" comes to mind.

But just over a decade later two astronauts managed to land and return safely. In another couple of years the country was bored by the concept; I remember watching a later mission blasting off in a quarter of the TV screen while a football game filled the rest.

Neal and Buzz took their lunar waltz 39 years ago. Even after four decades manned access to space is still fraught with danger and uncertainty. The costs are still astronomical. And America's only human-rated launch vehicle will go into mothballs in just two years.

It took just a decade to go from no space capability, through early manned launches, to a successful moon landing. Ironically if Orion meets all of its schedule goals, which seems unlikely given the routine overruns that plague big government contracts, it'll take 15 years of development and test to repeat Apollo 11's, by then, fifty-year-old accomplishment.

How did Kennedy's mandate succeed, especially given the primitive technology of the era? How did armies of mostly young engineers invent the spacecraft, the web of ground support infrastructure, and the designs of the missions themselves in such a short time?

The rich history of space exploration has been well-told in many books. My favorites include Carrying the Fire: An Astronaut's Journeys, by Michael Collins, the lonely pilot who stayed in orbit during Apollo 11's descent. Of all of the astronaut autobiographies this is the most literate and finely crafted.

Then there's Apollo 13 by Jeffrey Kluger and James Lovell. Lovell, of course, was the commander of that ill-fated mission. Though sometimes seen as a disaster, 13 strikes me as a Macgyver-esque tale of success. After big explosion 200,000 miles from Earth, in the depths of space, their consumables running out, the crew and engineers on the ground cobbled together fixes that brought the mission home. That the fragile craft could sustain such major damage is a testimony to everyone involved in the program.

But these books, like most of the others on the market, are largely about the pilots who rode into space. They're shy on technical details. Just how did we navigate to the moon, anyway? Whizzing along at 2000 miles/hour it moves about a quarter pi radians around the Earth in the three days of flight. How did managers grow the program from a dream to hundreds of thousands of workers and $25 billion in so few years?

Enter Apollo: The Race to the Moon by Charles Murray and Catherine Bly Cox. This page-turner (at least for geeks) juxtaposes a picture of a Mercury capsule being haphazardly trucked to the pad, swaddled in mattresses, with the monstrous crawler-transporter ponderously moving a Saturn V. But sadly, the book is out of print and used editions on Amazon start at a whopping $74.

One might think that after 40 years this subject would be exhausted, but Bill Schweber of Planet Analog recently alerted me about a new book called Digital Apollo by David Mindell. It falls into the ranks of the best Apollo books for geeks.

It's not unusual for a space book to reach back to the story of the X-15 or even Chuck Yeager's X-1 record-breaking flight, but in Digital Apollo Mindell starts with the story of the Wright Brothers. Yes, they are generally credited with inventing the airplane, but he focuses only on their experiments with wing warping and other forms of control. For the Wrights's pioneering work succeeded largely because they invented ways to steer a flying machine in three axis. And control, particularly as manifested in the Apollo Guidance Computer, is one of the two overarching themes of the book.

Digital Apollo starts with an initially somewhat boring investigation into test pilot psychology. Those flyers formed the cadre from which most of the astronauts were selected. Why should a modern reader care about some jet jockies' inflated self-esteem? Mindell skillfully weaves that discussion into the nature of aircraft stability, instrumentation, and then into his second major theme: Who should control the vehicle? The pilot, or a computer?

He avoids the obvious question of why, if the automata are so good, we bother to include a person in the spaceship. That essentially political and philosophical debate belongs elsewhere. But it certainly goes to the nature of being human and what exploration means in the virtual age. Mindell does end with a captivating description of the 2004 meeting of the Explorer's Club where famous folk like Aldrin, Piccard (not the one of Star Trek fame) and Hillary (not the one in the news) listened to Pathfinder's Stephen Squyres give a talk about exploring Mars while comfortably seated in an air-conditioned lab.

At the very first meeting of the Society of Experimental Test Pilots in 1957 some attendees expressed fear that they would be automated out of a job. Throughout the book it becomes clear that much of the engineering of the various spacecraft was dominated by the tension between traditional rocketeers, who felt everything should be automated, and the pilots. Some Mercury astronauts wanted to fly the rocket right off the pad.

If control is just keeping a needle centered on a gauge why stick a heavy human in the control loop? But those with the Right Stuff want to be in control, to take heroic action, and those desires impacted the space program in the 60s and still do today.

The USSR believed in automation and their authoritarian culture brooked no interference from mere cosmonauts. Gagarin, the first man in space, was blocked from his spacecraft's manual controls by a combination lock whose code was in a sealed envelope!

Despite Mercury being the smallest and most primitive of the Apollo triptych, pilots of those first six missions had the least control of all. All important operations were automatically sequenced. System failures forced Scott Carpenter to assume manual control. He saved the mission, but various delays pushed Aurora 7 some 250 miles off course. Some saw this as a success for those advocating the need for the pilot in the loop; others, loudly, blasted him for poor performance.

The astronaut office rebelled at Mercury's spam-in-a-can design. On Gemini each mission's dynamic duo still had only systems management responsibility on launch, but were an integral part of the control loop in orbit. Armstrong and Scott saved their Gemini by reading a magnetic tape with reentry procedures into the computer.

The hero had been redefined as a boot loader.

About a hundred fascinating pages into the book Mindell begins to look at Apollo, and specifically the Apollo Guidance Computer (AGC), which was, other than software, virtually the same as the LM Guidance Computer. Recognizing the difficulty of controlling a spacecraft from the Earth to Lunar orbit, and thence down to the surface, NASA let the very first Apollo contract not to the spacecraft vendor, but to MIT's Instrumentation Lab for the AGC.

Oddly, everyone forgot about the code. The statement of work for the AGC said this about software: "The onboard guidance computer must be programmed to control the other guidance subsystem s and to implement the various guidance schemes." That's it. Lousy requirements are not a new problem.

Another phenomenon that's not new: once management realized that some sort of software was needed, they constructed a procedure called a GSOP to specify each particular mission's needs. But in practice GSOPs were constructed from the code after it was written, rather than before.

If Doxygen had been around they could have saved a lot of time.

Flash memory hadn't been invented. Nor had EPROM, EEPROM or any of the other parts we use to store code. For reasons not described in the book (primarily size) traditional core memory wasn't an option, so the AGC stored its programs in core rope. Programs were "manufactured" - the team had to submit completed tapes three months before launch so women could weave the programs into the core ropes that were then potted. Once built it was extremely difficult to change even a single bit.

The code was written in an interpreted language called BASIC, though it bore no similarity to the BASIC produced by Dartmouth. Listings of the AGC code (http://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/apollo/public/archive/1701.pdf) show that even in the 60s software engineers used random names for labels, like KOOLADE and SMOOCH, the latter suggesting perhaps too many hours at the office.

There was never enough memory (32K of program space). Then, like now, developers spent an inordinate amount of time optimizing and rewriting code to fit into the limited space. Consider that a tiny 32K program controlled the attitude indicator, all thrusting and guidance, a user interface, and much more.

And what was the user interface? Parts of the UI included windows (not on a desktop, but those glass things) that were etched with navigational info. The computer prompted the astronaut with angle data; he looked out through the markings and designated new targeting parameters to the computer. Did the computer have an UI. or did the pilot? Who was driving what? The pilot was just another component in the system.

Mindell paints the moon flights as a background patina to the guidance theme. An entire chapter is devoted to Apollo 11, but other than describing how the astronauts used the stars to align the inertial reference platform the author ignores the flight to and from the moon. The chapter covers the landing sequence in detail - some might say excruciating detail but I was riveted. Most remember the 1201 and 1202 alarms that caused great concern. It turns out that Aldrin wanted to leave the rendezvous radar (which was only needed to seek out the command module) turned on in case an emergency abort was needed. Program management signed off on this change, but no one told the software engineers. Data from the radar demanded computer attention that was in short supply, so it flashed the 1201 and 1202 codes indicating "I'm getting overwhelmed!" The AGC rejected low priority tasks but ran the landing activities correctly. That's pretty slick software engineering.

A single chapter covers the descent and landing phases of the next five missions (Apollo 13 is left out as it never entered lunar orbit). Many are familiar with the abort switch failure, later traced to a blob of solder, that could have ended 14's mission. That's described brilliantly, but the software patch is glossed over in a short barrage of words that left me feeling the author really didn't understand the fix.

This book is short on deep technical details about the AGC. For more on that see Journey to the Moon: The History of the Apollo Guidance Computer by Eldon C. Hall. But it provides a fascinating look at the human side of engineering, and how pilot demands (think "customer demands"), both reasonable and not, effect how we build systems. Those inputs, Mindell thinks, largely shaped the design of the Space Shuttle. He quotes astronaut Walter Cunningham about splashing down in the sea in Apollo: "[the crew were recovered] by helicopter like a bag of cats saved from a watery grave." And Cunningham's take on Shuttle: "[it] makes a smooth landing at the destination airport and the flight crew steps down from the spacecraft in front of a waiting throng in a dignified and properly heroic manner."

One wonders in this age of digital control of everything what pilot/computer tradeoffs will be made on the Orion spacecraft.