Why most project estimates suck…and how Monte Carlo simulations can make them better

missed deadlinesHave you ever been part of a project that was late and over budget? I’d be surprised if you haven’t. We humans are famously bad at estimating the future, and project planning is heavily dependent on our ability to estimate the future. Most of us are optimists and some of us are pessimists, but very, very few of us are realists by nature. Monte Carlo simulations can be useful in our estimation process to help us become more realistic about our estimates, and that realism can significantly improve our ability to deliver results more in line with expectations.

We generally recognize our inability to accurately estimate large projects in one chunk, so we break them up into smaller milestones that are easier to estimate. While the work breakdown process is good, the confidence it gives us in our estimates can lead to larger problems. We don’t ask ourselves often enough how accurate we think those estimates are before stringing them together to determine project due dates. If we did, the conversation might go like this:

“How accurate do you think these milestone estimates are?”

“Pretty accurate. We certainly spent a lot of time discussing them and comparing them to past projects.”

“OK. But if you had to put a number on it, would you say they are 100% accurate?”

“Well, let’s not get crazy. I can’t be sure they’re 100% accurate.”

“So put a number on it. How confident are you that they’re accurate?”

“I still feel pretty good about them. I’d say conservatively that I’m at least 90% sure.”

At this point, we’re about to discover some pretty major problems with our assumptions. We typically string together a number of these milestones, which are dependent on each other, and call them the critical path. The end of the critical path is the project due date.

But if we’re only 90% confident our estimates for each milestone are correct, the likelihood of missing our date is pretty high. Let’s say we have five major milestones in our critical path, and we’re 90% sure each is accurate. To determine the probability that all five will come in as expected, we have to multiply .90 x .90 x.90 x .90 x .90. Even with these high confidence rates, we’re now looking at about a 59% chance of hitting our due dates and a 41% chance of missing them. And that’s with only five milestones and really high (and probably unwarranted) confidence in our estimates. The numbers only get worse from here.

So we start missing deadlines and inevitably either pump more money into the effort or start cutting scope. Our original business case and ROI justification for the effort are now inaccurate because it’s going to cost more and produce less benefits. Sound familiar?

Monte Carlo simulations can help us get a better handle on the probabilities of actually delivering on our timeline and budget estimates. Just as I previously demonstrated using Monte Carlo simulations for sales forecasting, a simulation focused on project estimates can essentially become a “what if” model and sensitivity analysis on steroids for project planning. Basically, the model allows us to feed in a limited set of variables about which we have some general probability estimates and then, based on those inputs, generate a statistically valid set of data we can use to run probability calculations for the entire project.

Great. So now we know how likely we are to miss our timeline and budget. So what?

Once we have a more realistic view our our project timeline and budget, we can do far more effective planning. We can develop contingency planning with full knowledge of the likelihood of needing any particular contingency. Having a better sense of potential budget increases or scope decreases in advance of the project start date will help us make better decisions about starting the project to begin with.

We’ll also be able to better plan our needs from other groups in the corporation who might be involved with the final project but not directly involved in the project. For example, we might need to fit a new product launch campaign into an already packed marketing schedule. Will new site functionality require training for customer service? We’ll need to plan time to pull agents off the phones for their training. Setting expectations with these external groups will greatly enhance at least the internally perceived success of our effort. And that certainly counts for something.

Why go through all this complication? Let’s just take all the estimates we get from the team and double them. That should help get ensure we stay within the timeline.

The “double the estimates” approach is one I’ve seen used before. While it does help create timelines that won’t be exceeded, overestimation can also cause problems. Any coordination with external teams will still be a problem if we end up needing them before we originally planned. And over-allocating time, resources and budget can drive up opportunity costs and limit our ability to produce meaningful results over time.

Monte Carlo to the rescue

I created a free, sample Monte Carlo simulation you can download for use in project planning. It illustrates on a small scale some of the possibilities that can occur with even a minor project. We see that even a five milestone effort with 85% confidence in the estimate of each milestone is expected to be more that 20% overdue. But we can also get a sense of the probabilities of various timelines and use it to refine overall estimates.

By understanding the probability of various delivery dates and project budgets, we can better plan scope, business models and contingency plans. We can better coordinate with other teams who will play a part in the ultimate success of the project once it’s complete. In short, we can become realists and, as a result, deliver much better business results.

What do you think? Would this sort of tool help in your planning? What other methods have you used to set better expectations and plan more accurately?

Blinded By Certainty

blindfoldedIn reality, very little in our lives is absolutely certain. We can be certain the sun will rise in the east and set in the west. We can be certain death will follow life. And we can be pretty darn certain Steve Jobs will wear a black turtleneck and jeans at his next public appearance.

But we’re certain about a lot more things than we should be.

A recent University of Michigan study by Brendan Nylan and Jason Reifler shows that the more certain we are about particular ideas or situations the more we become blind to facts that discredit our certainty. In fact, in many cases opposing facts are not just ignored but actually strengthen our prior beliefs.  A recent Boston Globe article provides an excellent summary of the research.

From the article:

Most of us like to believe that our opinions have been formed over time by careful, rational consideration of facts and ideas, and that the decisions based on those opinions, therefore, have the ring of soundness and intelligence. In reality, we often base our opinions on our beliefs, which can have an uneasy relationship with facts. And rather than facts driving beliefs, our beliefs can dictate the facts we chose to accept. They can cause us to twist facts so they fit better with our preconceived notions. Worst of all, they can lead us to uncritically accept bad information just because it reinforces our beliefs. This reinforcement makes us more confident we’re right, and even less likely to listen to any new information.

Both the research and the article focus primarily on our political viewpoints, but while reading I couldn’t help but think of people I’ve come across in the business world who were unbelievably certain about their viewpoints based on information or experiences that seemed less than obvious to me. I immediately thought of dozens of people, and I bet you’re thinking of many such people now.

In fact, it was so easy for me to think of other people that fit the bill that I couldn’t help but think the man in the mirror was not immune to this universal human fallacy.

In my experience in the business world, we often assume with undue certainty that past experiences will reflect future possibilities. We say things like, “We tried that before and it didn’t work” or “I know what our customers want.” While our past experiences are extremely valuable and are very important for informing future decisions, we simply don’t have enough of them to blindly ignore changes in circumstances, timing and other variables that could significantly alter results for a new effort.

So how do we overcome our natural instincts in order to make better business decisions?

  1. Be aware of the problems with certainty
    You’ve read this far, so maybe you’re awareness is already active. I know that I am reassessing all the things I “know” to try to truly separate what is fact and what is assumption. I very much value all my experience, and I know I make better decisions because of what I’ve seen and heard along the way. But I want to make doubly sure that assumptions I make based on past experiences are tested and validated before I turn them into absolute fact.
  2. Actively seek alternate points-of-view
    In my experience, the combination of multiple experiences provides a much more solid foundation for decision making than basing decisions on singular past experiences. Techniques I’ve used, like The Monkey Cage Sessions, are based on the incorporating viewpoints from people in different functional areas and levels of the organization. While it’s acceptable to discount data or opinions that are in opposition to a decision I might make, I want to be sure I’m not simply rationalizing opposing information or viewpoints solely because they are different from my biases.
  3. Envision alternate scenarios
    I addressed this some in a previous post, “Obscure and pregnant with conflicting meanings”, where I discussed a technique I called “Scenario Imagination.” I’ve since read an excellent interview with Daniel Kahneman and Gary Klein where they detail a similar and better technique they call “pre-mortem” (which is also a better name than mine). Whenever we make decisions, we have a tendency to assume our decisions are going to produce the best possible results. These pre-mortem techniques have us imagine worst case scenarios to try to dissect potential problems before they occur.
  4. Be flexible and plan for contingencies
    Once we admit we’re not 100% certain, we can move forward with plans that are flexible and able to react to changing conditions. To be clear, I’m not saying we should just be wishy-washy and not make clear decisions. What I’m saying is that we should be open to new facts and be sure we have created an environment that allows us to change course when warranted.

If we’re aware of our certainty biases and take active steps to address them, I believe we can significantly improve our decision-making in our businesses.

What do you think? Upon self-examination, have you turned beliefs into facts in your mind? How would you suggest addressing these biases? Or, do you think is all a load of hooey?

Retail: Shaken Not Stirred by Kevin Ertell


Home | About