Think Design : The Overconfidence Effect

I’m going to make a prediction. If I’m right, you have to read the rest of this post. Deal? Okay, here goes…

If you work in the games industry, you have been on a game project where at least one of these things happened:

  1. You had to cut features.
  2. The game went over schedule.
  3. The game went over budget.

I knew you’d still be reading. Almost every project in our industry suffers from one or more of these outcomes because of something called the Overconfidence Effect.

Here’s a challenge for you: The Sydney Opera House began construction in 1959. The original estimate stated that it would be completed in 1963 for a total cost of about $7 million. When do you think it was actually completed, and how much do you think it actually cost? Try to be accurate. I’ll give you the answer later.

The overconfidence effect says that we overestimate our ability to predict and forecast. And not just by a little.

A group of experiments was conducted where participants were asked to estimate the numbers of various things, such as the number of doctors in the phone book or the number of eggs produced in the US each year. People could choose any range of numbers they wanted, with the goal being that they should be within two percent of the actual value. In other words, they should be almost certain that the correct answer was going to fall within the range they chose. On average, people were over forty percent off the mark.

It’s important to clarify, though: the overconfidence effect isn’t about our ability to estimate. It is about how confident we are in the estimates we make. It is the gap between what we know and what we think we know.

And it usually applies to complex things – the simpler something is, the easier it is for us to forecast. It’s easy for us to forecast that we’ll have our shoes on in about fifteen seconds. It’s tougher to forecast how long it will take for us to get ready and drive downtown to meet at the restaurant. But you can bet that when we give our friend the estimate (‘See you in forty-five minutes’), we’re confident that the estimate is accurate. Little do we know we forgot to wash the shirt we were thinking of wearing so need to think of a backup, and the elevator in our condo is out so we need to take the stairs. An accident has the highway really backed up, and when we finally get downtown we can’t find parking.

This same thing happens when we try to estimate the amount of time or money we’ll need to make a game. It happens when we estimate how long it will take to overcome technological challenges, or nail down our visual style. It happens when we think our first playable will be fun and everything will be polish from there.

14 years in development. Worth the wait?

 

Further studies have shown that as we gain more information, we become more confident in our forecasts, despite the fact that our accuracy doesn’t generally improve. This means that experts are even more susceptible to the overconfidence effect than novices. It also makes so-called experts potentially dangerous, because they tend to announce their incorrect forecasts with authority. Because they sound more credible, more people believe them and trust their guesses. In the game industry, this means that experienced producers and execs are just as likely as everyone else to make estimation errors.

Pessimists tend to have more accurate estimates, because they are more conservative in their predictions, but they still fall prey to the overconfidence effect.

So what can you do when planning your next project?

1) Try to look at previous actuals if you have them. They can give you a good guide for which aspects of your project tend to require more iteration time for your time, or places where you needed to spend more money than anticipated. Keep in mind that data about what happened last week can become actuals to use this week.

2) Consider how far off your estimates have been in the past. Take a look at each team member’s estimates and actuals, and adjust their new estimates accordingly. Perhaps one team member consistently over-estimates the amount of time needed to complete a task, while another person’s estimates need to be multiplied by three to approach accuracy.

3) Be pessimistic. Assume that you will need more time and money than you initially think to cover unforeseen events, and account for it up front by adding buffers. Ideally, this will put you in a position where you will finish early and/or under budget.

4) On any new project, keep in mind that you will almost certainly encounter tech issues, some features won’t be as easy to implement as you hoped, the initial design might not be awesome right away, and the team will get tired as the project goes on (and on, and on).

So do your best to account for these outcomes, but prepare to encounter one or more of the problems mentioned at the beginning of this article, no matter how carefully you try to estimate. Try to embrace the uncertainty. After all, one of the exciting aspects of being a game developer is encountering these unexpected situations and overcoming them. Kind of like playing a game.

Oh, and the Sydney Opera House was completed in 1973, with a total cost of $102 million. Ten years late and fourteen times over budget. Whoops.


Ryan Donaldson teaches Business of Games at VFS Game Design