Chasing Developer Productivity Metrics
I was listening to an episode of the “Western Devs” podcast on a plane the other night (I really like this podcast, by the way — give it a listen). The subject of this particular show was the idea of developer productivity and the measurement thereof. At the risk of playing spoiler, the discussion digressed in interesting fashion without necessarily taking a serious run at conclusions about these measurements.
But asking for a definitive measurement of developer productivity is a tall order. I would actually even consider the attempt to be quixotic, though not for the reason alluded to early in the show (the idea that it might be categorically impossible to measure). I think there are any number of ways, some even credible, to measure productivity for developers. I think the trouble arises from the notion that they could be applied outside of a narrow context and also that it’s especially important how productive a developer is.
Let me return to that claim a little later, though. First, I want to talk about The Organization.
What We Want from the Organization
It’s important, at this point, to dispense with the euphemisms. “Productivity,” in a business context, is a measure of efficiency, which is a comparative ability to deliver the same amount of stuff in less time (or for less money, though, in wage-land, time and money are like energy and matter in special relativity). So productivity is really “worth” for labor.
For developers, then, it becomes more personal. Monetary worth is a bean counter distinction, and they don’t really know worth, we tell ourselves. They don’t understand what Alice brings to the table when she goes around performing targeted refactorings that make everyone more productive, and they don’t recognize how important Bob is to the team’s morale, even if he doesn’t, himself, deliver a lot of code.
So we create situational, alternate definitions of productivity that reflect our ethos and emotional attachment to what we do. We re-couch productivity as a way of describing the meritocracy that we feel ought to exist within the company. And it is this very tendency that leads us to discuss endlessly “what does developer productivity even mean and how do we measure it?” Our definitions are aspirational rather than practical. We want an orderly world and we want the organizations for which we work to radiate fairness toward us.
But the organization has other ideas. The organization provides individualized feedback on our productivity/efficiency (i.e. our performance) in the form of performance reviews. And boy do we ever misunderstand these.
Performance Reviews as a Corporate Cost Savings
I would argue that the main purpose of the corporate performance review, as an institution, is to save money. This might seem counter-intuitive to a lot of the players in the process who probably think of it as a way to reward loyalty and performance. Savings? That seems like a silly goal for the thing that results in raises being handed out. But consider the following bullets that I’ll explain in a little more detail below.
- Reviews are the primary vehicle for justifying small or nonexistent raises (middling to poor performance) while…
- Inoculating the company against lawsuits regarding unfair pay practices.
- Reviews offer low level managers the perk of power to dole out additional money to their favorites and withhold it from redheaded step children (for managers that are into this sort of thing).
- Reviews serve as a sounding point to guard against turnover and attrition costs.
- Reviews are (usually) infrequent enough to allow nearly indefinite postponement of pay increases while the reviewee considers them to be in the bag.
Think of most corporate review processes to which you have been exposed. They’re usually stuffy, weird affairs where your manager uncomfortably reads off a sheet of paper and soberly informs you how well you have embodied the company’s virtue of “integrity” over the last year. The meeting will sometimes offer vaguely programming-related feedback, but it’s usually more or less an evaluation of you as a human, in very antiseptic, business terms.
It will often be the case that you’re given an atta-boy/girl and told that you’re getting a better than average, 3.5% pay increase and you’re really, truly, no-foolin’ getting better at everything. Next year when your “customer focus” score is a little better, you’ll probably be ready to go from Software Engineer III to Software Engineer IV.
What’s happening here is basically that the company is deferring a significant (i.e. promotion-accompanying) pay increase as long as possible, according to a budget-oriented, pre-canned narrative. You’ll be a software engineer III for as long as they can reasonably keep you there. The language of the review will include the specific notion of productivity and performance, but will evaluate you according to extremely malleable, fuzzy criteria.
The fuzzy formalism serves to justify small pay increases to you, but it also serves to offer up plausible deniability and a pay matrix in case people become litigious. The review process ensures that no one is advancing too quickly or slowly without a paper trail of cause. And it is during this process that double checks will happen for legal land mines, like letting salaries get out of whack for the wrong age groups or protected classes or anything else that confers grounds for lawsuits.
But there’s an element of give and take here as well. Savvy managers will be monitoring your reaction to gauge for undue disappointment. If the manager considers you valuable and worth keeping, she may intervene on your behalf, should you become too glum, preventing the need for later making counter-offers and diving saves when you interview elsewhere.
And, speaking of the manager, reviews are a nice perk for some managers, depending on their level of enjoyment of being the bagman with the company’s money. Some people truly get kicks out of holding that kind of power, and, in cases where this applies, the company offers this intangible perk. Any perk adds to the total package, putting slight downward pressure on wages. And, this perk is subtly intoxicating, since it gives a low level manager the simulated experience of having a massive enough fortune to change lives – the “Scroodge Simulation,” if you will.
The upshot of all of this is that the performance review, writ large, is a means of corporate cost control. And, in the world of software developers, its calibrated to err on the side of turnover rather than overpayment. This makes the corporate concept of evaluating developer productivity a rather depressing one.
Back to Basics
I have a consulting practice that includes a mashup of writing blog posts, IT management consulting, and app dev. It’s just me under the LLC umbrella, but I’m starting to take on subcontractors in a limited capacity. Let’s consider what evaluating productivity of developers might mean in my world.
Overwhelmed, I look to sub-contract some writing and some app dev. I need a blog post about web services and I need some code written that invokes a web service. If I contract these things out to someone, I basically measure value as I would with a vendor — “did they deliver what they promised?” It’s fairly binary.
This would apply, more or less, whether the gig was fixed bid or hourly. I’d have a price in mind, and I would evaluate people I was paying against that price (not that I’d be unmovable on it – if they could demonstrate that I was underestimating, I’d be happy to revise). So, working for 100 hours on a blog post and charging me 10K for it would cause me to surmise that this was not a very productive person and I would look elsewhere.
Productivity in the world of contracting happens on two sides. If you reliably deliver what you promise, when you promise, within budget, I will consider you to be productive. You may consider yourself to be productive if you cut the time it took you to write the posts in half. In the former case, I’d evaluate you well because I can rely on you and, in the latter case, you’d evaluate yourself well because you’ve increased your earning efficiency.
Note one thing here. Efficiency is in the eye of the person paying (whether in dollars or variable rate labor). And that is so important – everything else is dross.
Illegibility and Fuzziness of Organizations
In a large enterprise, who is paying for developer 24601’s labor? If you’re honest, this is strictly unknowable. Millions of customers give the organization billions of dollars, which are paid out to tens of thousands of people.
The answer you’ll hear at organizations is that some upper level manager, VP, or whatever is paying for it. After all, that person is in charge of department X and has an annual budget of 40 million dollars, which includes the wage labor of our hero, developer 24601.
But that guy isn’t really paying, unless you’re willing to accept an awful lot of moral hazard baked into the fiber of the corporate entity. That guy’s not cashing in his 401K to pay those developers. He’s not heading over to the bank to take out a personal loan. Rather, he’s entrusted with some company money – he’s a manager and not an owner, which means he’s just managing the resources and making proxy decisions on behalf of those with real money to spend.
His only skin in the game is the relatively low stakes prospect of his own advancement. He is efficient if he produces more milestones in the enterprise’s ERP software for the same amount of money, as compared to his peers. That’s how he’s evaluated and it’s what he cares about.
So money pours into the company and it drifts its way into various budgets managed by various proxies in nice suits. From there, a tiny fraction of it trickles into developer 24601’s bank account every couple of weeks. A chain of people exist between the developer and the bigwig manager, who may not even be acquainted. So massive numbers of customers pay for something, and a chain of people are left to ask for things from their subordinates and, in some fashion, attempt to weigh the value provided to them by that subordinate compared to the others. But this value provided is not really the value to them – it’s value to their advancement prospects, if they hit their numbers.
You see, somewhere between my “am I getting what I want for my money from this writer/coder” and the organization’s monetary black hole, within whose event horizon no one knows what happens, the concept of value and efficiency becomes so abstract as to be pointless.
It is for this reason that I say measuring developer productivity in organizations is quixotic. It could, perhaps, be measured in some form or another, but no one would have any skin in the game (except, perhaps, for the consultant that came up with the measure) – no one would have a path to value. The measure that would, actually, matter in your day to day is “how much do I help my boss advance her career and how aware is she of that?” She’s your window to value. And someone else is hers, on up the chain, and into oblivion.
We like to talk about measuring our productivity because we like to chase the meritocracy and the order that it implies. We like to demonstrate mastery. But, here’s the thing.
If productivity is going to be measured at your organization, it will most likely translate into how effectively you help your superiors navigate organizational politics. But, if a measure were to be created that normalized things across the board, it would almost certainly be used to depress your wages. So let’s focus on measuring something else, like mastery or how many of your fellow techies you’re able to help up their skills.
Interesting read. I would like to point out that you are making Productivity practically synonymous with Efficiency. They are quite different as measurements. Efficiency is a metric that keeps the output constant: For 1 blog post I only charge $100. Productivity keeps the cost constant: For $100 I can deliver 2 blog posts. Given the task at hand for the developer, either metric might be considered. If the task needs to be executed quickly, then Efficiency is the driver. If a lot of code needs to be executed in a short timeframe, then Productivity is needed. If you stack up… Read more »
That’s a good point — I was pretty loose with the two terms. I didn’t give it a ton of thought as I was writing because I was more interested in the framework for arriving at the equation, if you will. For instance, if we were talking about rating developers based on the general idea of them physically moving around, I’d probably have animatedly thrown out terms like “distance covered” and “velocity” as interchangeable. That’s not correct to do, but if the problem is how to track their movements, then which physical property or properties you hold constant and which… Read more »
Good call on Western Devs. Just took in a couple during my lunch walk. I think I’ll keep this one in my subscription feed.
Thanks.
Happy to help. Maybe one day when I’m light on content, I’ll post the podcasts I check in on.
Interesting. I wonder if this is why a certain kind of developer prefers startup environments: if your thesis that what matters is “how much do I help my boss advance her career and how aware is she of that?” holds, and my boss is the co-founder/CTO/etc, then her career advancement is basically making the business as a whole successful enough to either retire rich or relax into a sinecure. That’s a very direct, unabstracted way of being evaluated: my boss’s desires align well with mine, because I want to make the product better (by writing good software), and making the… Read more »
I think there’s a definite appeal (for many, anyway) to working somewhere that has little indirection between what they do and visible results for the company. In my book and my general philosophical examination of the corporation as an entity, I think this is because organizations scale terribly. They hit a certain number of employees (probably not a very high number) and it easier to measure political contributions than economic/value-based ones.
Definitely agreed on the conceptual framework. Fewer people means more alignment on goals and more obvious goals, and that’s attractive to a lot of people, I think.
You should also remember to spend some time devising good and sensible measurements, so that your measurements actually mean something. I’ve just read an article on how and when measure development productivity… http://www.shorelabs.com/blog/2016/07/26/how-or-weather-to-measure-software-development-productivity