What Metrics Should the CIO See?
Editorial Note: I originally wrote this post for the NDepend blog. You can check out the original here, at their site. While you’re there, give NDepend a try — download it and see if your code falls in the dreaded Zone of Pain.
I’ve worked in the programming industry long enough to remember a less refined time. During this time, the CIO (or CFO, since IT used to report to the CFO in many orgs) may have counted lines of code to measure the productivity of the development team. Even then, they probably understood the folly of such an approach. But, if they lacked better measures, they might use that one.
Today, you rarely, if ever see that happen any longer. But don’t take that to mean reductionist measures have stopped. Rather, they have just evolved.
Most commonly today, I see this crop up in the form of automated unit test coverage. A CIO or high level manager becomes aware of generally quality and cadence problems with the software. She may consult with someone or read a study and conclude that a robust, automated test suite will cure what ails her. She then announces the initiative and rolls out. Then, she does the logical thing and instruments her team’s process so that she can track progress and improvement with the testing initiative.
The problem with this arises from what, specifically, the group measures and improves. She wants to improve quality and predictability, so she implements a proxy solution. She then measures people against that proxy. And, often, they improve… against that proxy.
If you measure your organization’s test coverage and hold them accountable, you can rest assured that they will improve test coverage. Improved quality, however, remains largely an orthogonal concern.
The CIO’s Leaky Abstraction
The issue here stems from what I might call a leaky organizational abstraction. If the CIO came from a software development background, this gets even more thorny.
Consider that a CIO or high level manager generally concerns himself with organizational strategy. He approves and monitors budgets, signs off on major initiatives, decides on the fate of applications in the application portfolio, etc. The CIO, in other words, makes business decisions that have a technical flavor. He deals in profits, losses, revenues, expenses, and organizational politics.
Through that lens, he might look at quality problems across the board as hits to the company’s reputation or drags on the bottom line. “We’re losing subscribers due to these bugs that happen at each roll out. We estimate that we lose $10,000 more each month in revenue.” He would then pull the trigger on business solutions: hiring consultants to fix this problem, realigning his org chart, putting off milestones to focus on quality, etc.
But if he dives into the weeds, he’s shedding a business person’s hat for a techie’s. “Move over architects,” he says, “I know how you can fix this at the line level. I call it ‘automated test coverage’ and I order you to start doing it.” In a traditionally organized corporate structure, the CIO begins doing the job of folks in his organization at his peril.
