On Metrics

Summary: It's all about people, process, and good data.

Good friend George Dinwiddie has a blog about software development, and his latest posting (http://blog.gdinwiddie.com/2011/07/11/process-standards/) caught my attention. George talks about the balance between people and process. He faults organizations that find the most productive developer, and then tries to clone him across the team by duplicating whatever processes he uses.

Some teams will make a change, perhaps adopting a new agile approach, and then loudly broadcast their successes. But without numbers I'm skeptical. Is the result real? Measurable? Is it a momentary Hawthorne blip?

Developers will sometimes report a strong feeling that "things are better." That's swell. But it's not engineering. Engineering is the use of science, technology, skilled people, process, a body of knowledge - and measurements - to solve a problem.

Engineers use metrics to gauge a parameter. The meter reads 2.5k ohms when nominal is 2k. That signal's rise time is twice what we designed for. The ISR runs in 83 usec, yet the interrupt comes every 70. We promised to deliver the product for $23 in quantities of 100k, and so have to engineer two bucks out of the cost of goods.

Some groups get it right. I was accosted at the ESC by a VP who experienced a 40% schedule improvement by the use of code inspections and standards. He had baseline data to compare against. That's real engineering. When a developer or team lead reports a "sense" or a "feeling" that things are better, that's not an engineering assessment. It's religion.

Metrics are tricky. People are smart and will do what is needed to maximize their return. I remember instituting productivity goals in an electronics manufacturing facility. The workers started cranking out more gear, but quality slumped. Adding a metric for that caused inventory to skyrocket as the techs just tossed broken boards in a pile, rather than repair them.

Metrics are even more difficult in software engineering. Like an impressionistic painting they yield a fuzzy view.

But data with error bands is far superior to no data at all.

Published July 11, 2011