On Measurements

By Jack Ganssle

Published in Embedded Systems Programming, June, 2002

On Measurements

I bet you don't know - quantitatively - what your productivity is, measured in lines of code per hour or any other units. (Most teams average a few hundred lines of code per month per programmer, measured from program outset to shipping).

I bet you don't know - quantitatively - your effectiveness at meeting schedules. (80% are delivered late). Or the cost of a new project (generally $15-$30 per line of code). Or how ship dates are exponentially related to code size (typically dates soar with the number of thousand lines of code raised to an exponent around 1.5 or more).

I bet you can't anticipate - quantitatively - bug rates in a new development. Yet bugs and poor quality are responsible for more late projects than any other factor. Typically, once a module compiles cleanly, expect about a 5% error rate.

Some developers subscribe to the belief that we're practitioners of a complex creative process that implicitly defies measurement. Like artisans building a sculpture we invent products from the void, building something from nothing more than our own cleverness. Did Da Vinci work to a schedule? It'll be done when it's done!

Balderdash.

MBA textbooks claim "if you can't measure it, you can't manage it."

Also balderdash.

If you're mass-producing widgets dozens of different measurements will indeed yield the information needed to optimize both productivity and quality. Likewise, financial controls are nothing more than extremely detailed measures of a company's performance. In both cases managers have a fiduciary responsibility to employ every possible tool to help the business thrive.

Wise engineering managers, too, must employ every reasonable tool to optimize development. Abdicate management and chaos will reign. For all but the smallest projects it's impossible to coordinate the efforts of multiple teams, and to keep individuals on-track, without using a reasonable set of metrics.

Metric Abuse

Unfortunately most of us run in terror when the new management team announces yet another round of metrics. Experience shows that the MBA's love to collect data! but they seem much less apt at interpreting it, and at effecting useful change from the measures.

My favorite dysfunctional measurement is the time card. In most organizations it's that dreary Friday-afternoon routine we remember at the last minute, when the paycheck police withhold the check till we turn in a completely filled-out timecard.

The motivation is getting paid; the "entry criteria" (as the process people would say) is filling in the time card. There's neither feedback nor closure. We could enter codes indicating vacation time or parties in Los Vegas (if there were an appropriate charge number); both would suffice for getting the paycheck.

We're all smart people, most of us went to college. Those lab classes taught us a lot of skills - none more useful than fabrication of data. So we exercise our creative powers in recreating more or less what happened during the week. The data is meaningless, of course. Few honest folks expect much accuracy on the timecards. Yet the ritual is repeated week after week in companies scattered across the globe.

We persist in this parody of precision, yet surely there's better approaches. Lawyers and others whose careers rest entirely on accurate billing, sometimes to 15 minute resolution, have tools that make gather such data both easy and accurate. It seems to me that either it's time to stop taking the data, or to use an alternative that's meaningful.

As a teenager I worked as a technician for an outfit which, in a desperate attempt to fix massive quality problems, mandated that every part we replaced in the production units go out for analysis by a lab to find the root causes of the failures. What if we erroneously replaced something that was, in fact, fine? No problem, management patronizingly assured us; they knew we wouldn't make such mistakes. We did. Since every replaced part must by definition be bad, it behooved us to ensure that even these mistakenly-changed devices had failed. I remember putting 110 VAC across a metal-can op amp hoping to fry the silicon!. Imagine my shock when the power blew a hole in the package! I never found out what root cause the lab assigned that failure to.

Silly measurements create total contempt from the development team for any metrics. If your organization is so broken consider some other, easy, measurements that can yield interesting data while getting people to think. For instance, the Aspirin Metric: place a big bottle of aspirin in a centrally-located area and log its rate of consumption. Or the Friday Beer Tab Metric: create a spreadsheet that logs the bar tab for the team's weekly gathering. Both approaches can tell a lot about burnout.

Managing strictly by measurement corrupts the data we're trying to gather. Heisenberg showed us that we can not know both the position and momentum of a particle; measuring one effects the other. The same is true in organizational metrics. A wonderful Dilbert shows the pointy-hair boss telling his tech writer that she'll be incented based on the number of words she writes, with points off for those deleted. "Don't want to reward the wrong behavior" the boss struts. In the next scene Tina maxes her pay, while minimizing productivity, by writing massive amounts of jibberish.

Years ago, when running a tool business, I noticed that rules. Shipments quickly increased! but quality fell. The imposition of more measures boosted quality. Inventory levels skyrocketed. The techs gave up repairing bad boards and simply stacked defective ones.

I learned that management by measurement is like squeezing a balloon smaller. For a while it works, but then an unexpected aneurysm pops out. We're simply not smart enough to anticipate all possible problems.

Why Measure?

Engineering managers take data for one of three reasons. The first is to reward or punish the participants.

As a young engineer working on a big job with an impossible schedule, my boss told me that if the unit was "done" (whatever that meant) by the show date I could accompany it to Europe. That sounded pretty good to me. The machine and I did get our overseas vacation. The unit barely worked, of course. My goals were satisfied while the company's were less so. Rewards have to be structured very carefully so both parties get exactly what they want.

Working for that company made the most dramatic theme park ride look sedate. Its fortunes rose and fell at a dizzying pace. When we went public I overheard an outside auditor say "I've seen every trick this outfit has pulled! but never all by one company." So it wasn't unusual for the boss to tell me that if we were late or over-budget on a new project I'd have to lay off a few people. That's a horrible way to measure one's performance. The threat of such punishment hanging over our heads drove the best people to headhunters and humiliated the rest.

Measures are a poor way to motivate people. Deming, the quality guru, found there are two kinds of motivators. The first, called "intrinsic" are those that come from within, from deep in our souls. Wanting do a good job. Feeling like part of a great team. Taking pride in our work.

He also studied "extrinsic" motivators: when the boss makes a silly demand, or imposes strict rules we don't understand or support. Deming found, to no one's surprise, that extrinsic motivators drive out the intrinsic ones. We comply unwillingly and drag our feet.

Or, we resist. In the 60s we learned that passive resistance is an effective way to toss sand in the machinery. Go limp when the cops bust you; they have to drag you away. The same thing happens in engineering. We all know those managers will be gone in two years, promoted or fired. Do nothing for a while and the boss and rules will change.

Measures are ineffective motivators. Take data for one reason only: to get insight into a product and process. To learn what is going on now, so we can improve it tomorrow.

Good Measures

Every time someone proposes a metric ensure that it will lead to greater insight and understanding. Ruthlessly abandon those that don't.

There are four components to any measurement: collecting, interpreting, presenting and acting on the data. Typically we think in terms of data collection and forget about the other three, equally important, steps.

Raw data is pretty, well, raw. Counting source code lines might make sense, but until it's normalized somehow (are you counting whitespace? Comments?) it's a nonsensical jumble of unrelated numbers. If counting bugs, what constitutes a bug? A violation of the firmware standard? We've got to interpret the data to make sense of it, and to remove randomness that comes from the collection process.

This is the communications age. We are the people who built it, who constructed the cellular and Internet equipment that facilitates an unprecedented amount of person-to-person interaction. Yet we're often the least capable users of this fantastic infrastructure. Communications is the key to career growth and to effective teamwork. Once the data makes sense we must present it to ourselves, our colleagues, our bosses or our customers. Presentation is more than a dreary recitation of facts and figures. It's a sales job. Breathe life into the  sterile numbers, derive and demonstrate the insight the figures give to the job at hand.

Many of us are reluctant or even terrified to stand up in front of a crowd. I was till my thirties, when I realized how career-limiting giving in to the fright was. So I decided to take some action and learn to give talks. The first few times were horrible, sweat-filled hours that left me ready to delve back into solo programming forever. But with practice it became easier, and eventually very enjoyable. If the thought of talking to a group leaves you fearful, or if you leave the crowd with glazed, bored, eyes, take some action. Fix the problem. Dale Carnegie and Toastmasters both have effective programs.

Finally, act on the data. Interpreted, presented numbers are useless unless we effect some change. Improve something. Run experiments. Tune the process and see what happens to the numbers.

Collect, interpret, present and act. Neglect any one of these four and the measurement is fruitless. None of us have time to waste. If you can't commit to all of these, don't do any.

Other Requirements

Measurements must be easy. Onerous data collection always fails. It's abandoned as soon as constant oversight lags.

Measurements must make sense today. Unfortunately metrics can be like government bureaucracies. Once imposed, even when used to understand a particular problem, they never go away. One case in point is a company faced with huge turnover. Various measurements indicated that the problem stemmed from bizarre program management procedures. Process changes all but eliminated defections, which plummeted and plateaued within months, yet years later they still took the same measures.

Maybe it makes sense to re-measure at much less frequent intervals, but wise management will impose a sunset law on every metric, holding each to review at frequent intervals.

Finally, measurements must be visible so people see what is going on. They need to see the value of action versus avoidance or even outright sabotage. It all comes down to people and their buy-in. Lose that, and everything collapses.

Conclusion

I was sailing on a friend's boat recently, going great guns in a decent breeze. In a quest for speed he put up the spinnaker. Foam flew, the wake grew, and Lee was elated at our apparent much greater velocity.

Having looked at the knotmeter both before and after flying the bigger sail I knew the facts: we weren't going any faster. Sure felt like it, but the boat was at hull speed and no amount of sail would accelerate her.

Without the numbers Lee was running by the seat of his pants, a notoriously ineffective way to do anything. What we feel is tempered by what we expect and want. Only quantitative data tells the truth.

But desiring to be a polite guest I kept my observations to myself. The knotmeter told the truth, but being unwilling to act of the information I needn't have bothered to look at it.