Burned by a Consultant

Summary: The source code was in limbo, somewhere on the consultant's machines.

A friend and I started a consulting company in 1981. The embedded age was quite young and our customers, even the technical types, were mostly bewildered by the fungible notion of hardware and firmware. We always presented them with a full documentation package, with source, schematics, etc. I suspect most of that wound up in dusty corners where it was later lost.

I've known Fred for over 20 years. We met through a business relationship; he bought quite a few of the tools my emulator business sold. We have a lot of parallel interests and quickly formed a friendship. Fred is a ton of fun, and we laugh a lot together.

The emulators he bought went to his consultants, a group of varied individuals over the years who had developed his communications product. There's now quite a bit of code in the thing. For the last dozen years "Joe" has been the point man; he has maintained and greatly extended the code, as, like all products, the requirements changed and grew over the years. Joe works remotely from his house 1500 miles from Fred's facility in New England.

Joe died.

He had developed cancer, and did tell Fred about it. Many months after the diagnosis. Mere weeks before he succumbed.

Any death is a tragedy, but business must go on. Fred discovered that over at least the last 18 months Joe hadn't been checking code into Fred's server, despite a contractual obligation to do so. His business will fail without the source code.

In a panic he visited the widow at Joe's house. In the twelve years they worked together Joe had always managed to deflect requests to visit his home office. Fred found a veritable pile of junk stacked in every room. They were hoarders. Much of the $75,000 of test equipment he had sent Joe was missing. or perhaps lost in the mess.

The code was, well, somewhere spread amongst five computers. One was password protected and no one knows what the password is; Fred is pretty sure the code to one of his boards is irretrievably locked up there. The other machines each had fragments of source in a jumble of directories with nonsensical names. There was no organization, and file names were inconsistent and cryptic. The widow was uncooperative and insisted that two of the machines didn't have Fred's code; after much persuasion, with a lawyer on the phone, she relented, and, fragments of source were also on those boxes. He sucked over 30 GB of files onto a USB drive, but isn't sure what is what.

Over the last 12 years Fred paid Joe $1.6 million to write and maintain this code. Now he's left with the metaphorical equivalent of a pile of shoe boxes of unknown files. He'll have to hire someone to make sense of it all; to figure out what comprises which versions of the product; to recreate a development environment and to figure out the build process. Fred is 70 and has his life savings wrapped up in the business. It's not clear at this point how things will play out.

No matter how established a relationship is, regardless of how close a friendship may be, when it comes to source code, follow Ronald Reagan's advice: trust but verify.

 When relying on consultants, establish an identical development at your facility and routinely make sure you can build the code. Check the binary against that supplied by the outsider. Yes, this is a pain. Don't do it and your business may fail.

 When a consultant is your entire engineering department, either examine the work products yourself or hire another to look at them from time to time. ensure that reasonable care is being taken with naming, directory structure, and all of the other nuances that quality code requires.

 We engineers are good at doing FMEAs to understand how products may fail; if you have a small operation like Fred's, run an FMEA on the business. One clear failure point is inadequate control of the source code, as well as poor backups of that code. I wrote (http://www.embedded.com/electronics-blogs/break-points/4436135/On-backups) about this recently.

We routinely do worst case analysis to ensure our creations will work over temperature extremes. The same should be done for the business.

Published November 20, 2014