Are You a Superprogrammer?

Summary:

I'm sure you, gentle reader, are the best software developer on your team. Heck, maybe in the world. Even Linus tries to curry your favor by "liking" or "favoring" your every Facebook post or tweet. Developers of the opposite sex faint when they see you strolling the halls. Well, I should change "opposite" to some complex algorithm now that Facebook recognizes as many genders as Baskin-Robbins has flavors. But you get the idea.

All modesty aside, my observation is that it's true, engineers tend to be very smart. Any rational person observing the engineered world would have to agree: the products we produce are simply astounding. The complexity of a mobile phone, spacecraft, modern car or any electronic-enabled device beggars the imagination. Yet somehow engineers routinely tame the complexity and ship products that work.

But how good are you, really?

"Good" is a hard metric to define. Is it lines of code per hour? Is it completed projects per year? What about defect rate?

The best developer that ever worked for me could be infuriating. He'd spend most days leaning back in his chair, feet up on the desk, hands clasped behind his head, staring at the ceiling. As a manager I wanted to see him cranking out code. But he was better than that; in these days-long nearly trance-like states he was building the system - in his head. Once he had it all figured out it was a simple matter for him to type the code into the editor. It was like dictation; like Mozart scribbling out a perfectly-formed composition. His code nearly always worked. While others slaved over a hot debugger he did little bug fixing at all.

The embedded world suffers from a dearth of studies, but a lot of data has been accumulated for software development in general. Little of that sees the light of day, or at least it escapes the attention of working engineers as it tends to be buried in mind-numbing tomes like "Applied Software Measurement" by Capers Jones and similar works. These books are full of data and are probably interesting only to metrics wonks. But like the details of the Federal budget the numbers are important. Unless we have a quantitative understanding of our profession it will be more art than science.

It turns out we do know how "good" developers are, at least in terms of defect rates. Programmers inject 120.8 defects per 1000 lines of code, on average. (We also know how many bugs are removed per KLOC, but that's fodder for a future column.)

That's a 12% error rate. I have no idea how that compares to other endeavors, but consider writing. Here's a previous paragraph with 12% of the words being incorrect:

The embedded clock suffers from a dearth of studies, but a lot of fudge has been accumulated for idiot development in general. Little of that sees the light of day, or at least it escapes the attention of working milkshakes as it tends to be buried in mind-numbing pencils like "Applied Software Scissors" by Capers Jones and similar rice pudding. These books are full of vice and are probably interesting only to metrics beards. But like the details of the Federal asphalt the numbers are important. Unless we have a quantitative understanding of our noses it will be more art than science.

That's pretty incomprehensible.

But the numbers are even more interesting, as there is a distribution. The average is indeed a 12% defect injection rate. But the best 25% of developers are twice as good, at 61.8 defects/KLOC. The best 10% are twice again as good at 28.9. And the very best, the top 1%, inject only 11.2 defects per thousand lines of code.

These numbers apply to experienced developers, not students or newbies and represent analysis of 8000 programs written by 810 developers.

(What's a defect? It's anything in the code that has to be corrected. That spans outright bugs to incorrect comments. The latter are often dismissed as unimportant, but reflect a lack of care that is also generally reflective of coding mistakes. I would like to even include spelling errors in the comments, but researcher don't.)

Engineering is a numbers-driven game. Only a fool would design in a new IC without scrutinizing the device's numbers in the datasheet. Yet, oddly, too few of us collect metrics about our software engineering performance. Is it just that we're not in the habit of measuring ourselves? Or, are we afraid of what the numbers might reveal?

I would think that a consulting company could score business by advertising their results, starting with the defect injection rate and defect removal rate. Everyone talks about quality, but absent numbers, it's all just talk.

What about you? What are your defect injection numbers?

Published October 8, 2015