Prediction Markets for Software Estimates
The ongoing kerfuffle over the “No Estimates” movement is surreal to me. So before I get to prediction markets for software estimates, I’ll discuss the surreal a bit. Instead of attempting to describe it, which, I can only imagine, would be meta-surreal, I’ll make use of an allegory of sorts.
No Wedding Estimates
Imagine that wedding planners have long struggled with an important issue. For years and years, their clients have asked them to help pick a date on which it would not rain so that they could have outdoor weddings. And, for years and years, their predictions have been spotty at best, resulting in irate clients, wet formal wear, and general heartburn.
In all that time, wedding planners have pored over weather patterns and diligently studied farmers’ almanacs. And yet, the predictions did not substantially improve. It got so bad that a group of upstart wedding planners met one year in the mountains and authored a document called “The Contingency Manifesto.” This resulted in an important advance in wedding planning for outdoor weddings: reserving a backup plan not dependent on the weather.
And yet, not all was fixed. People still wanted outdoor weddings, and continued to be disappointed when it rained, contingency notwithstanding. They still demanded wedding planners help them figure out months and months in advance whether or not it would rain on a particular day. And, this is understandable after all — weddings are important.
Quite recently, a group of wedding planners emerged under the hashtag #NoOutdoorWeddings. Their message was simple: “we can’t predict the weather, so have your wedding inside and stop whining.” The message was, of course, music to the ears of frustrated wedding planners everywhere. But some planners and most clients balked. “How can you tell the clients not to have weddings outside? They’re the ones paying, so it’s our obligation to facilitate their wishes!”
This schism, it seemed, was irreparable. And surreal.
- Why is it necessary for wedding planners all to agree? Can’t the ones that don’t want to deal with weather contingency just not do it and the ones who want to can?
- Why can’t people figure out that trying to predict a chaotic system like the weather is a fool’s errand?
- Why would you ask a wedding planner to make a prediction that could easily be influenced by his own interest?
- Frankly, if one person is dumb enough to ask another to predict the weather on a day 9 months from now, and the other person is dumb enough to do it, can’t we just agree that the two deserve each other and the inevitable lawsuit?
How Estimation Actually Works Today
Now, I realize that this isn’t a perfect parallel, but it serves the purpose of illustrating how surreal the arguments about software estimation seem to me. I can’t really wrap my head around people taking estimates seriously, let alone making financial decisions as if the estimates were reliable. That’s like asking a physicist to size up a roulette wheel right after the ball is tossed, guess where it will land, and betting heavily on it. That’s not strategy, and if it makes you money, it’s because you got lucky.
And think about what happens when someone asks you for an estimate. Be honest with yourself. Someone asks, “so, assuming that we can get you access to the database you need in the first 3 weeks and that they don’t change the web page you need to scrape more than once a month, what do you think we’re looking at here in terms of weeks? 12? 14?” What goes through your head?
If you’re being honest, it’s probably this:
Holy crap, I have no idea! Okay, okay, think. Man, I dunno. I want to say 5 years, but she’ll laugh at me. Maybe she wouldn’t laugh if I said 2 years. Right, she’ll yell at me. Hmm… I did a project like this once, except we used XML files instead of a database and we didn’t scrape anything or have a GUI. Crap, that was actually nothing like this. I feel like, honestly, I could do it in like 5 weeks if everything went well, but last time I estimated like that, I got fired when it took 32 weeks.
“I think it’ll be about 32 weeks!”
Crap! Why did I blurt that out?
“But, that’s just a really rough estimate. Really. Rough.”
At this point, if either of you thinks that you should plan an expensive launch party in 33 weeks, I have some magic beans that might interest you.
We’ve been getting weather forecasts for decades, and they still break down so completely after 10 days that we don’t even bother making them. We’ve been making estimates about software for decades, and they continue to be similarly useless. So shouldn’t it come down to this: get better or don’t bother?
And it seems to me that people just can’t help demanding estimates and others just can’t help providing them, notwithstanding a well intentioned group of people no longer willing to pretend that the emperor of estimation is wearing clothes (and I do see value in socializing the reality that estimates ranging further than a few weeks are pretty much useless). So if that’s the case, why not get better?
Software construction is chaotic in the same way that weather is chaotic (and that, say, building construction is not chaotic). And, do you know what else is chaotic? Markets, such as stock and futures markets. And, as an added comparison bonus, a lot of people insist that they can predict them and some manage to fool themselves and others.
But that’s individuals trying to game the market. Where it gets a little more interesting is when you consider the valuations of the market itself. That is, if I tune in to listen to Jim Cramer scream stock market advice on me and decide that IBM is about to go out of business, I’ll be wrong and lose my money. But, if the entire market decides that IBM is worthless, it will be right.
Prediction Markets FTW?
So here is what I propose, and bear in mind that this is an extremely raw concept. I thought it up while jogging this morning and I am, by no means, an economist, so it will have holes in it. It’s just meant to be a conversation starter.
If you commission a piece of software, don’t bother asking the developers that will be writing it for an estimate. If you want an estimate, you pay for it by flooding a software project prediction market with some money. When you do this, you also need to furnish enough information about the project to allow traders to form informed opinions about when the software might be done. Traders then buy positions in the form of, say, weeks that the project will be completed. If I thought your project would be done in about 3 months, I’d pay money to hold the first week of 2016.
As the project wound on, information would continue to be furnished to the market about progress, requirements changes, different design approaches, etc. As this happened, trading would be open to anyone interested. If the project seemed to be dragging, I could sell my position in week 1 of 2016 and buy shares for later. Or, if I wanted to gamble on a longshot, I could buy positions in early December for pennies.
There are obviously some things to figure out. How do you get enough information to the market without risking the company’s position if information is proprietary? And you’d also need to work out a way to prevent developers on the project from having access to the market for their project to prevent “insider trading/influence.” I’m sure there’s more.
But there are some definite advantages to this approach.
- Incents interested parties to surface information about risks and efficiencies, whereas “ask the people doing it” incents risks to be buried and efficiencies exaggerated.
- Rewards successful estimation and punishes unsuccessful estimation (current system does not — usually providing comically optimistic estimates is how you win contracts).
- Replaces a single, conflicted-interest estimator with an entire population of informed, neutral estimators.
- Introduces a mild deterrent to requesting estimates — to get good ones you have to pay. Companies would learn to do without whenever possible.
- The culture of browbeating developers for estimates and then bludgeoning them with the results would go away.
Like holes and things that would need to be addressed, there are probably a bunch of other advantages I’m not thinking of. That said, this has a long way to go before it goes from “crack-brained scheme” to “let’s run an experiment,” and farther still to “this could actually work.” But I think it’s at least as good an idea as continuing to have shouting matches over whether bad estimates are better than no estimates or not.
Here’s a sampling of databases we use when making estimates in the enterprise IT domain
There are other databases in NASA, DOD, DOE, and other federal agencies, as well as ones in construction, power stations, biopharma
I like the concept of gathering the data for regression analysis and comparison. In fact, in a lot of analyses I do, I start gathering data first before I have a real firm sense of what I want to analyze. It’s just always, “I’ll probably want this later.” A couple of questions come to mind: (1) Is the primary purpose of these databases cataloging properties of the projects and properties of outcomes for the purpose or regression analyses and comparison? (2) I looked around on some of the sites, but didn’t see this (I may have missed it) — how… Read more »
You’re confusing making informed projections is hard with it is impossible. Your ““I think it’ll be about 32 weeks!”” is the number one problem in what goes wrong with ‘estimates’ it’s totally nonsensical to respond to this question. The only correct answer is “i’ll get back to you. will you be available on short notice to answer questions i have regarding this?” That’s a wild guess, not a valid estimate. To answer how long something takes you need to break it down into discrete blocks, compose an architecture of using the blocks. Then you need to estimate each block, each… Read more »
I agree with all of Chris’s points here, following a process such as this does tend to produce better numbers than “about 32 weeks”, and although it’s a time consuming process spent doing something that isn’t producing any software, for projects that are unlikely to have a good ROI, it could be useful by getting these projects cancelled before they start. Better to lose the price of doing some estimates than the price of doing a runaway project that was never going to be profitable. There is one very big caveat however. Many projects change so much over the course… Read more »
This is a nice summary of my own take on the matter. Even with the BEST possible project plans and risk mitigation strategies, I’m not sure that I believe estimating deliverable dates for these projects is any better than estimating share prices for market securities. Even with perfect information about the past, it is really, really, really hard to to predict the future (which is why I offer the references to markets and weather for comparison).
I think people grossly over estimate the complexity of their software and vastly under estimate engineering. I tried to find large engineering projects that were delivered on time and google wasn’t helpful, it did lead me to look up stuff about the hoover dam. I couldn’t really find the words “delayed” or “exceeded budget” referring to building something that will be around 10,000 years. I did find an article about the hoover dam bridge https://www.asme.org/engineering-topics/articles/construction-and-building/hoover-dam-bridge-top-10-facts Although bureaucratic and contractual delays and a major equipment mishap added years to the timeline, the project met its $240-million budget — $114 million of… Read more »
A project like the Hoover Dam is a classic example of one where a waterfall approach can work. Requirements for a dam or bridge are not likely to significantly change over the course of the project. No one is going to try to add new features to dam once all the contracts have been signed and construction has been started.
(most) Software projects are different. Most of the problems and changes can’t be anticipated up front; at least some percentage of the effort to do BDUF will be wasted effort.
You don’t think things changed during it? You don’t think some contractor was supposed to deliver X by Date and it didn’t happen and they had to adapt to the reality of it?
A contractor delivering late is not a change in requirements.
That’s still a change to the project. It’s not different. It doesn’t matter why an activity changes, it potentially changes the project network. Many changes can be absorbed by a project that is built with a correct tolerance of slack in it. If a project has none to little slack or changes happen to the critical path then the project is delayed.
Slack can be built in to accommodate people getting sick, having to change resources, etc. You can not anticipate changes in requirements, or put slack in the project budget for that, unless you are just doing a straight fudge of the numbers.
I don’t understand how you can say they are not different. here is a absolutely huge difference between these types of changes.
You don’t anticipate someone getting sick, you deal with it. You don’t anticipate requirement changes, you deal with it. You don’t anticipate a contractor failing to deliver, you deal with it. You don’t know the impact if you can’t analyze the project network before and after the change. Great example of what happens if you blindly change the project. Steve is out sick. Phil is not working on the project, but was planned to be involved soon. You’re able to get Phil to cover for Steve and handle X. Activity Y is highly dependent on X and it was planned… Read more »
The cost is absolutely minimal. It should cost about 3% of the project to determine this. I’d much rather spend $30,000 to find out a project will cost $3 million dollars and i can’t afford it. Than spend $1 million building something and having to cancel it from running out of money. Without this type of project planning you can’t ever know the 2 most important questions to a business “how much will it cost?” and “when will it be done?” The inalienable need for change during a project’s construction raises the importance of this even further. Having a detailed… Read more »
Which estimation process do you follow? Do you estimate in man days/story points etc?
I estimate small discrete units of a system in man-weeks. Then by plotting a network diagram of the activities based on their dependencies I determine the critical path of the project. The critical path is the the duration of the project. This becomes a projection and not an estimate. I then analyze the float/slack time of the project and apply mathematical formulas to determine the risk of the project, risk being a non-linear value for the chance of project failure. Failure being not delivered on time and on budget.
Is this a system that you’ve devised yourself or a known process that you’ve read about?
IDesign and Juval Lowy’s master classes. He formulated all of this basing it off the work of David Parnas and many others.
Which life cycle model are you using?
Can you elaborate on what you’re looking for as an answer here?
Are you using Scrum/XP/Pure Waterfall/PRINCE/CMMI etc?
All of those are really orthogonal to architecture and project design, except perhaps XP. I’m not familiar with prince
The software development life cycle determines when architecture and project design is done, and whether it is all done up front, or done in pieces.
Everyone I know who follows a “no estimates” approach is Agile and usually doing Continuous Delivery.
And absolutely nothing in agile or continuous deliver specifically precludes having concrete architecture plans or a project design. You can align activities to sprints no problem. The only downside to using ‘kanban’ is you have to normalize activities to a standard person and can never plan that the best person for the work will do it. The downside to using sprints is it increases waste from shoving in arbitrary dates that don’t inherently exist into the project (but this is how a sprint works inherently, or you constantly push work / pull work from the next sprint completing defeating the… Read more »
How many developers work on a typical project in your company? How many of them provide the estimates?
Previous org I generally had 2-3 teams of 3-5 developers working on 2-3 projects at a time. I made all estimates. Past year i’ve been doing consulting for large organizations, again been responsible for estimates.
So you are able to provide better estimates than those doing the delivery. And if they don’t deliver on time and budget then they have clearly failed.
So you are able to provide better estimates than those doing the delivery Assuming an architect knows their team well? Definitely. If you don’t know your team, then more effort needs to be done. Once again this goes back to creating well defined blocks that are a finite amount of work. Nothing should offer infinite amounts of work. And if they don’t deliver on time and budget then they have clearly failed. Incorrect. Not completing a project on time and on budget is solely the responsibility of the architect. In a project with an appropriate level of risk, no individual… Read more »
Suppose you gave estimates for a project and then it became apparent to you that the developers weren’t taking the project very seriously or putting very much effort in. If at the end of the project the schedule had been missed, would you still hold your hands up and say “its my fault, my estimates were wrong”?
Being an architect doesn’t mean you walk away from a project once it’s planned. It means being involved and tracking its status atleast weekly (honestly it takes only a few minutes a day with Earned Value Analysis and a simple spreadsheet or have a project manager do it). If progress trend lines are not heading as projected, it’s our job to figure out why. Are other stakeholders pulling employees to work on different projects. Are developers sandbagging the project and working on personal projects or just reading facebook all day. Were the estimates just globally unreasonable? All of these scenarios… Read more »
It sounds like your role is project manager as well as architect. Would you agree with this?
Fair enough, but it’s also fair to say that if there’s a hat for it, I likely do wear it or have worn it at some point.
I have only worked for one company that has the solutions architect job title, but I’ve seen a few different incarnations of that role. When I joined, the role was very similar to what I do now and think of as a senior developer role. The architect would belong to a delivery team and do some work to ensure code written was reusable, but spent 90%+ time writing code. Then a solutions architecture team was created and the architecture team was moved away from delivery. It was told told it was not allowed to write any production code anymore. They… Read more »
And anyone questions why most projects are doomed before the first line of code is written. A successful project doesn’t start with i’ll give you 3 of these and 2 of those and this and that.
I’m assuming this is a good link to info, for my reference? http://www.idesign.net/about
Yes and the models were created by them. But for the most part pretty simple. You deem each activity a weight based on its risk: 0 float aka critical path high risk, very low float just treat it as critical (if it overruns it becomes critical), some float medium risk, tons of float low risk. Then compute the weighted average of the project. For the most part it reaffirms reality if a project has very little float it is very risky. If a project is developed by 1 person it is maximum risk as every activity is critical and every… Read more »
I like the distinction between projection and estimate.
Are the mathematical models you’re referring to proprietary, or is this something that you could point me at and I could read about?
“Without this type of project planning you can’t ever know the 2 most important questions to a business “how much will it cost?” and “when will it be done?”” I think most would argue that even WITH this type of planning, you can’t answer these questions. Or that you can answer it (because you’re required to by the business or client), and then are still wrong a good percentage of the time. I’m sure there are a handful of people out there who are really good at estimating, and maybe you’re one of them. But in my experience, most estimates… Read more »
If your estimates are continuously wrong you’re not breaking things down into manageable units. If you have a block that is 2 weeks, 3 weeks of magnitude shouldn’t be taking 6 weeks, 8 weeks. If it is, you still haven’t broke it down properly. Also if you ever give a time frame without knowing a project’s critical path you’re blindly guessing and you’ll always be wrong.
I can’t really respond outside of the general arguments against estimates, which I’m sure you’ve heard before.
I’ve worked with a lot of people who claim to be good at estimating, but I’ve never met anyone that actually was. I believe it’s theoretically possible to do a decent estimate for a smaller project that has detailed requirements, but this is not the norm in my experience. Even then, I question the value of putting work into a detailed estimate over giving something more high level (e.g. S/M/L/XL).
The t-shirt sizing you’re referencing is a great unit for a single activity. It is not something you can measure a system with, a system that will be comprised of easily several dozen activities across multiple people. To understand a system you must have a project’s network of dependencies. Building all the right pieces but in the wrong order could be the difference of numerous months and millions of dollars.
And to your statements never met anyone that was good. It’s because they don’t estimate, they guess, blindly guess.
“Building all the right pieces but in the wrong order could be the difference of numerous months and millions of dollars.”
Much more likely in waterfall than in agile.
So if an estimate was wrong, it must have been a guess? Estimates are good, but your way is the only way that works?
Only a method that uses engineering to arrive at answers can be consistently reliable. There’s a reason bridges aren’t built with alchemy. Sure maybe sometime alchemy will build a bridge but don’t ask me to be the one to stand under it or on top of it.
If it’s not your way, it’s alchemy. Got it.
Any method that doesn’t use repeatable engineering practices. Doing stuff randomly, repeatedly, is not an engineering process.
For what it’s worth, your answer of “I’ll get back to you” is what I actually give clients in such situations. *Refusing* to provide any kind of estimate has always struck me as an odd position. If it’s a guess, I tell whoever is asking quite frankly, “this would just be a guess.” If they want my guess, I’ll give it to them. My tale here is one of how I’ve witnessed all too many “estimations” go over the course of my career. I once saw a consulting firm estimate a 6 figure project *down to the dollar* after about… Read more »
In all seriousness a 6 figure project is almost about as small as they can come. This is basically a small development team working for a couple of months. If the problem is incredibly well defined this can absolutely be done in 2 days. It always takes more time to figure out what a client truly wants than it does to figure out how to build it optimally. Determining how much a project will cost down to the dollar is easy too. It’s just the summation of the cost of all the people working on it plus any fixed costs… Read more »
From what I recall, the project was around 600K, and after their estimate down to the dollar, they eventually delivered it for about 1.5 million. But, let’s say that a project is 150K with labor at 150 per hour and no travel or equipment costs. This means 1000 hours and, for a small project, let’s say it’s 4 people so it’s 250 hours per, for a touch over 6 weeks per. This means for an estimate down to the dollar to make sense, you’d spend 2 days up front and know exactly how many hours (or half hours or whatever… Read more »
Wow, this sure turned into a lively discussion! My apologies — I’ve been buried in work this week and have not been anywhere near able to keep up 🙁
I feel bad for how far i keep making footer go down.