Chasing Developer Productivity Metrics
I was listening to an episode of the “Western Devs” podcast on a plane the other night (I really like this podcast, by the way — give it a listen). The subject of this particular show was the idea of developer productivity and the measurement thereof. At the risk of playing spoiler, the discussion digressed in interesting fashion without necessarily taking a serious run at conclusions about these measurements.
But asking for a definitive measurement of developer productivity is a tall order. I would actually even consider the attempt to be quixotic, though not for the reason alluded to early in the show (the idea that it might be categorically impossible to measure). I think there are any number of ways, some even credible, to measure productivity for developers. I think the trouble arises from the notion that they could be applied outside of a narrow context and also that it’s especially important how productive a developer is.
Let me return to that claim a little later, though. First, I want to talk about The Organization.
What We Want from the Organization
It’s important, at this point, to dispense with the euphemisms. “Productivity,” in a business context, is a measure of efficiency, which is a comparative ability to deliver the same amount of stuff in less time (or for less money, though, in wage-land, time and money are like energy and matter in special relativity). So productivity is really “worth” for labor.
For developers, then, it becomes more personal. Monetary worth is a bean counter distinction, and they don’t really know worth, we tell ourselves. They don’t understand what Alice brings to the table when she goes around performing targeted refactorings that make everyone more productive, and they don’t recognize how important Bob is to the team’s morale, even if he doesn’t, himself, deliver a lot of code.
So we create situational, alternate definitions of productivity that reflect our ethos and emotional attachment to what we do. We re-couch productivity as a way of describing the meritocracy that we feel ought to exist within the company. And it is this very tendency that leads us to discuss endlessly “what does developer productivity even mean and how do we measure it?” Our definitions are aspirational rather than practical. We want an orderly world and we want the organizations for which we work to radiate fairness toward us.
But the organization has other ideas. The organization provides individualized feedback on our productivity/efficiency (i.e. our performance) in the form of performance reviews. And boy do we ever misunderstand these.
Performance Reviews as a Corporate Cost Savings
I would argue that the main purpose of the corporate performance review, as an institution, is to save money. This might seem counter-intuitive to a lot of the players in the process who probably think of it as a way to reward loyalty and performance. Savings? That seems like a silly goal for the thing that results in raises being handed out. But consider the following bullets that I’ll explain in a little more detail below.
- Reviews are the primary vehicle for justifying small or nonexistent raises (middling to poor performance) while…
- Inoculating the company against lawsuits regarding unfair pay practices.
- Reviews offer low level managers the perk of power to dole out additional money to their favorites and withhold it from redheaded step children (for managers that are into this sort of thing).
- Reviews serve as a sounding point to guard against turnover and attrition costs.
- Reviews are (usually) infrequent enough to allow nearly indefinite postponement of pay increases while the reviewee considers them to be in the bag.
Think of most corporate review processes to which you have been exposed. They’re usually stuffy, weird affairs where your manager uncomfortably reads off a sheet of paper and soberly informs you how well you have embodied the company’s virtue of “integrity” over the last year. The meeting will sometimes offer vaguely programming-related feedback, but it’s usually more or less an evaluation of you as a human, in very antiseptic, business terms.
It will often be the case that you’re given an atta-boy/girl and told that you’re getting a better than average, 3.5% pay increase and you’re really, truly, no-foolin’ getting better at everything. Next year when your “customer focus” score is a little better, you’ll probably be ready to go from Software Engineer III to Software Engineer IV.
What’s happening here is basically that the company is deferring a significant (i.e. promotion-accompanying) pay increase as long as possible, according to a budget-oriented, pre-canned narrative. You’ll be a software engineer III for as long as they can reasonably keep you there. The language of the review will include the specific notion of productivity and performance, but will evaluate you according to extremely malleable, fuzzy criteria.
The fuzzy formalism serves to justify small pay increases to you, but it also serves to offer up plausible deniability and a pay matrix in case people become litigious. The review process ensures that no one is advancing too quickly or slowly without a paper trail of cause. And it is during this process that double checks will happen for legal land mines, like letting salaries get out of whack for the wrong age groups or protected classes or anything else that confers grounds for lawsuits.
But there’s an element of give and take here as well. Savvy managers will be monitoring your reaction to gauge for undue disappointment. If the manager considers you valuable and worth keeping, she may intervene on your behalf, should you become too glum, preventing the need for later making counter-offers and diving saves when you interview elsewhere.
And, speaking of the manager, reviews are a nice perk for some managers, depending on their level of enjoyment of being the bagman with the company’s money. Some people truly get kicks out of holding that kind of power, and, in cases where this applies, the company offers this intangible perk. Any perk adds to the total package, putting slight downward pressure on wages. And, this perk is subtly intoxicating, since it gives a low level manager the simulated experience of having a massive enough fortune to change lives – the “Scroodge Simulation,” if you will.
The upshot of all of this is that the performance review, writ large, is a means of corporate cost control. And, in the world of software developers, its calibrated to err on the side of turnover rather than overpayment. This makes the corporate concept of evaluating developer productivity a rather depressing one.
Back to Basics
I have a consulting practice that includes a mashup of writing blog posts, IT management consulting, and app dev. It’s just me under the LLC umbrella, but I’m starting to take on subcontractors in a limited capacity. Let’s consider what evaluating productivity of developers might mean in my world.
Overwhelmed, I look to sub-contract some writing and some app dev. I need a blog post about web services and I need some code written that invokes a web service. If I contract these things out to someone, I basically measure value as I would with a vendor — “did they deliver what they promised?” It’s fairly binary.
This would apply, more or less, whether the gig was fixed bid or hourly. I’d have a price in mind, and I would evaluate people I was paying against that price (not that I’d be unmovable on it – if they could demonstrate that I was underestimating, I’d be happy to revise). So, working for 100 hours on a blog post and charging me 10K for it would cause me to surmise that this was not a very productive person and I would look elsewhere.
Productivity in the world of contracting happens on two sides. If you reliably deliver what you promise, when you promise, within budget, I will consider you to be productive. You may consider yourself to be productive if you cut the time it took you to write the posts in half. In the former case, I’d evaluate you well because I can rely on you and, in the latter case, you’d evaluate yourself well because you’ve increased your earning efficiency.
Note one thing here. Efficiency is in the eye of the person paying (whether in dollars or variable rate labor). And that is so important – everything else is dross.
Illegibility and Fuzziness of Organizations
In a large enterprise, who is paying for developer 24601’s labor? If you’re honest, this is strictly unknowable. Millions of customers give the organization billions of dollars, which are paid out to tens of thousands of people.
The answer you’ll hear at organizations is that some upper level manager, VP, or whatever is paying for it. After all, that person is in charge of department X and has an annual budget of 40 million dollars, which includes the wage labor of our hero, developer 24601.
But that guy isn’t really paying, unless you’re willing to accept an awful lot of moral hazard baked into the fiber of the corporate entity. That guy’s not cashing in his 401K to pay those developers. He’s not heading over to the bank to take out a personal loan. Rather, he’s entrusted with some company money – he’s a manager and not an owner, which means he’s just managing the resources and making proxy decisions on behalf of those with real money to spend.
His only skin in the game is the relatively low stakes prospect of his own advancement. He is efficient if he produces more milestones in the enterprise’s ERP software for the same amount of money, as compared to his peers. That’s how he’s evaluated and it’s what he cares about.
So money pours into the company and it drifts its way into various budgets managed by various proxies in nice suits. From there, a tiny fraction of it trickles into developer 24601’s bank account every couple of weeks. A chain of people exist between the developer and the bigwig manager, who may not even be acquainted. So massive numbers of customers pay for something, and a chain of people are left to ask for things from their subordinates and, in some fashion, attempt to weigh the value provided to them by that subordinate compared to the others. But this value provided is not really the value to them – it’s value to their advancement prospects, if they hit their numbers.
You see, somewhere between my “am I getting what I want for my money from this writer/coder” and the organization’s monetary black hole, within whose event horizon no one knows what happens, the concept of value and efficiency becomes so abstract as to be pointless.
It is for this reason that I say measuring developer productivity in organizations is quixotic. It could, perhaps, be measured in some form or another, but no one would have any skin in the game (except, perhaps, for the consultant that came up with the measure) – no one would have a path to value. The measure that would, actually, matter in your day to day is “how much do I help my boss advance her career and how aware is she of that?” She’s your window to value. And someone else is hers, on up the chain, and into oblivion.
We like to talk about measuring our productivity because we like to chase the meritocracy and the order that it implies. We like to demonstrate mastery. But, here’s the thing.
If productivity is going to be measured at your organization, it will most likely translate into how effectively you help your superiors navigate organizational politics. But, if a measure were to be created that normalized things across the board, it would almost certainly be used to depress your wages. So let’s focus on measuring something else, like mastery or how many of your fellow techies you’re able to help up their skills.