Prediction Markets for Software Estimates

The ongoing kerfuffle over the “No Estimates” movement is surreal to me.  So before I get to prediction markets for software estimates, I’ll discuss the surreal a bit. Instead of attempting to describe it, which, I can only imagine, would be meta-surreal, I’ll make use of an allegory of sorts.

No Wedding Estimates

Imagine that wedding planners have long struggled with an important issue.  For years and years, their clients have asked them to help pick a date on which it would not rain so that they could have outdoor weddings.  And, for years and years, their predictions have been spotty at best, resulting in irate clients, wet formal wear, and general heartburn.

In all that time, wedding planners have pored over weather patterns and diligently studied farmers’ almanacs.  And yet, the predictions did not substantially improve.  It got so bad that a group of upstart wedding planners met one year in the mountains and authored a document called “The Contingency Manifesto.”  This resulted in an important advance in wedding planning for outdoor weddings: reserving a backup plan not dependent on the weather.

And yet, not all was fixed.  People still wanted outdoor weddings, and continued to be disappointed when it rained, contingency notwithstanding.  They still demanded wedding planners help them figure out months and months in advance whether or not it would rain on a particular day.  And, this is understandable after all — weddings are important.

Quite recently, a group of wedding planners emerged under the hashtag #NoOutdoorWeddings.  Their message was simple: “we can’t predict the weather, so have your wedding inside and stop whining.”  The message was, of course, music to the ears of frustrated wedding planners everywhere.  But some planners and most clients balked.  “How can you tell the clients not to have weddings outside?  They’re the ones paying, so it’s our obligation to facilitate their wishes!”

This schism, it seemed, was irreparable.  And surreal.

• Why is it necessary for wedding planners all to agree?  Can’t the ones that don’t want to deal with weather contingency just not do it and the ones who want to can?
• Why can’t people figure out that trying to predict a chaotic system like the weather is a fool’s errand?
• Why would you ask a wedding planner to make a prediction that could easily be influenced by his own interest?
• Frankly, if one person is dumb enough to ask another to predict the weather on a day 9 months from now, and the other person is dumb enough to do it, can’t we just agree that the two deserve each other and the inevitable lawsuit?

How Estimation Actually Works Today

Now, I realize that this isn’t a perfect parallel, but it serves the purpose of illustrating how surreal the arguments about software estimation seem to me.  I can’t really wrap my head around people taking estimates seriously, let alone making financial decisions as if the estimates were reliable.  That’s like asking a physicist to size up a roulette wheel right after the ball is tossed, guess where it will land, and betting heavily on it.  That’s not strategy, and if it makes you money, it’s because you got lucky.

And think about what happens when someone asks you for an estimate.  Be honest with yourself.  Someone asks, “so, assuming that we can get you access to the database you need in the first 3 weeks and that they don’t change the web page you need to scrape more than once a month, what do you think we’re looking at here in terms of weeks?  12?  14?”  What goes through your head?

If you’re being honest, it’s probably this:

Holy crap, I have no idea!  Okay, okay, think.  Man, I dunno.  I want to say 5 years, but she’ll laugh at me.  Maybe she wouldn’t laugh if I said 2 years.  Right, she’ll yell at me.  Hmm… I did a project like this once, except we used XML files instead of a database and we didn’t scrape anything or have a GUI.  Crap, that was actually nothing like this.  I feel like, honestly, I could do it in like 5 weeks if everything went well, but last time I estimated like that, I got fired when it took 32 weeks.

“I think it’ll be about 32 weeks!”

Crap!  Why did I blurt that out?

“But, that’s just a really rough estimate.  Really.  Rough.”

At this point, if either of you thinks that you should plan an expensive launch party in 33 weeks, I have some magic beans that might interest you.

Recognizing Chaos

We’ve been getting weather forecasts for decades, and they still break down so completely after 10 days that we don’t even bother making them.  We’ve been making estimates about software for decades, and they continue to be similarly useless.  So shouldn’t it come down to this: get better or don’t bother?

And it seems to me that people just can’t help demanding estimates and others just can’t help providing them, notwithstanding a well intentioned group of people no longer willing to pretend that the emperor of estimation is wearing clothes (and I do see value in socializing the reality that estimates ranging further than a few weeks are pretty much useless).  So if that’s the case, why not get better?

Software construction is chaotic in the same way that weather is chaotic (and that, say, building construction is not chaotic).  And, do you know what else is chaotic?  Markets, such as stock and futures markets.  And, as an added comparison bonus, a lot of people insist that they can predict them and some manage to fool themselves and others.

But that’s individuals trying to game the market.  Where it gets a little more interesting is when you consider the valuations of the market itself.  That is, if I tune in to listen to Jim Cramer scream stock market advice on me and decide that IBM is about to go out of business, I’ll be wrong and lose my money.  But, if the entire market decides that IBM is worthless, it will be right.

Prediction Markets FTW?

So here is what I propose, and bear in mind that this is an extremely raw concept.  I thought it up while jogging this morning and I am, by no means, an economist, so it will have holes in it.  It’s just meant to be a conversation starter.

If you commission a piece of software, don’t bother asking the developers that will be writing it for an estimate.  If you want an estimate, you pay for it by flooding a software project prediction market with some money.  When you do this, you also need to furnish enough information about the project to allow traders to form informed opinions about when the software might be done.  Traders then buy positions in the form of, say, weeks that the project will be completed.  If I thought your project would be done in about 3 months, I’d pay money to hold the first week of 2016.

As the project wound on, information would continue to be furnished to the market about progress, requirements changes, different design approaches, etc.  As this happened, trading would be open to anyone interested.  If the project seemed to be dragging, I could sell my position in week 1 of 2016 and buy shares for later.  Or, if I wanted to gamble on a longshot, I could buy positions in early December for pennies.

There are obviously some things to figure out.  How do you get enough information to the market without risking the company’s position if information is proprietary?  And you’d also need to work out a way to prevent developers on the project from having access to the market for their project to prevent “insider trading/influence.”  I’m sure there’s more.

But there are some definite advantages to this approach.

• Incents interested parties to surface information about risks and efficiencies, whereas “ask the people doing it” incents risks to be buried and efficiencies exaggerated.
• Rewards successful estimation and punishes unsuccessful estimation (current system does not — usually providing comically optimistic estimates is how you win contracts).
• Replaces a single, conflicted-interest estimator with an entire population of informed, neutral estimators.
• Introduces a mild deterrent to requesting estimates — to get good ones you have to pay.  Companies would learn to do without whenever possible.
• The culture of browbeating developers for estimates and then bludgeoning them with the results would go away.

Like holes and things that would need to be addressed, there are probably a bunch of other advantages I’m not thinking of.  That said, this has a long way to go before it goes from “crack-brained scheme” to “let’s run an experiment,” and farther still to “this could actually work.”  But I think it’s at least as good an idea as continuing to have shouting matches over whether bad estimates are better than no estimates or not.