Posts tagged: contingency planning

Why most project estimates suck…and how Monte Carlo simulations can make them better

missed deadlinesHave you ever been part of a project that was late and over budget? I’d be surprised if you haven’t. We humans are famously bad at estimating the future, and project planning is heavily dependent on our ability to estimate the future. Most of us are optimists and some of us are pessimists, but very, very few of us are realists by nature. Monte Carlo simulations can be useful in our estimation process to help us become more realistic about our estimates, and that realism can significantly improve our ability to deliver results more in line with expectations.

We generally recognize our inability to accurately estimate large projects in one chunk, so we break them up into smaller milestones that are easier to estimate. While the work breakdown process is good, the confidence it gives us in our estimates can lead to larger problems. We don’t ask ourselves often enough how accurate we think those estimates are before stringing them together to determine project due dates. If we did, the conversation might go like this:

“How accurate do you think these milestone estimates are?”

“Pretty accurate. We certainly spent a lot of time discussing them and comparing them to past projects.”

“OK. But if you had to put a number on it, would you say they are 100% accurate?”

“Well, let’s not get crazy. I can’t be sure they’re 100% accurate.”

“So put a number on it. How confident are you that they’re accurate?”

“I still feel pretty good about them. I’d say conservatively that I’m at least 90% sure.”

At this point, we’re about to discover some pretty major problems with our assumptions. We typically string together a number of these milestones, which are dependent on each other, and call them the critical path. The end of the critical path is the project due date.

But if we’re only 90% confident our estimates for each milestone are correct, the likelihood of missing our date is pretty high. Let’s say we have five major milestones in our critical path, and we’re 90% sure each is accurate. To determine the probability that all five will come in as expected, we have to multiply .90 x .90 x.90 x .90 x .90. Even with these high confidence rates, we’re now looking at about a 59% chance of hitting our due dates and a 41% chance of missing them. And that’s with only five milestones and really high (and probably unwarranted) confidence in our estimates. The numbers only get worse from here.

So we start missing deadlines and inevitably either pump more money into the effort or start cutting scope. Our original business case and ROI justification for the effort are now inaccurate because it’s going to cost more and produce less benefits. Sound familiar?

Monte Carlo simulations can help us get a better handle on the probabilities of actually delivering on our timeline and budget estimates. Just as I previously demonstrated using Monte Carlo simulations for sales forecasting, a simulation focused on project estimates can essentially become a “what if” model and sensitivity analysis on steroids for project planning. Basically, the model allows us to feed in a limited set of variables about which we have some general probability estimates and then, based on those inputs, generate a statistically valid set of data we can use to run probability calculations for the entire project.

Great. So now we know how likely we are to miss our timeline and budget. So what?

Once we have a more realistic view our our project timeline and budget, we can do far more effective planning. We can develop contingency planning with full knowledge of the likelihood of needing any particular contingency. Having a better sense of potential budget increases or scope decreases in advance of the project start date will help us make better decisions about starting the project to begin with.

We’ll also be able to better plan our needs from other groups in the corporation who might be involved with the final project but not directly involved in the project. For example, we might need to fit a new product launch campaign into an already packed marketing schedule. Will new site functionality require training for customer service? We’ll need to plan time to pull agents off the phones for their training. Setting expectations with these external groups will greatly enhance at least the internally perceived success of our effort. And that certainly counts for something.

Why go through all this complication? Let’s just take all the estimates we get from the team and double them. That should help get ensure we stay within the timeline.

The “double the estimates” approach is one I’ve seen used before. While it does help create timelines that won’t be exceeded, overestimation can also cause problems. Any coordination with external teams will still be a problem if we end up needing them before we originally planned. And over-allocating time, resources and budget can drive up opportunity costs and limit our ability to produce meaningful results over time.

Monte Carlo to the rescue

I created a free, sample Monte Carlo simulation you can download for use in project planning. It illustrates on a small scale some of the possibilities that can occur with even a minor project. We see that even a five milestone effort with 85% confidence in the estimate of each milestone is expected to be more that 20% overdue. But we can also get a sense of the probabilities of various timelines and use it to refine overall estimates.

By understanding the probability of various delivery dates and project budgets, we can better plan scope, business models and contingency plans. We can better coordinate with other teams who will play a part in the ultimate success of the project once it’s complete. In short, we can become realists and, as a result, deliver much better business results.

What do you think? Would this sort of tool help in your planning? What other methods have you used to set better expectations and plan more accurately?

“Obscure and pregnant with conflicting meanings”

We’ve all heard the cliché “hindsight is 20/20” a thousand times. And it’s pretty much true. It’s a lot easier to figure out the path to a particular event when you know the final outcome. But if “what happened” is something bad, determining the reason after the fact doesn’t change the negative event.

How can we do a better job finding those problems in advance of our next new strategy implementation, site redesign, store remodel or other big effort?

It’s worth digging a little deeper to better understand why our hindsight is so perceptive. One of the most famous cases of 20/20 hindsight comes from the investigation into the attacks on Pearl Harbor (although, we could also argue the investigation into 9/11 and the more recent Fort Hood shootings have many similarities). In her book Pearl Harbor: Warning and Decision, noted military intelligence historian Roberta Wohlstetter wrote “it is much easier after the event to sort the relevant from the irrelevant signals. After the event, of course, a signal is always crystal clear; we can now see what disaster it was signaling since the disaster has occurred. But before the event it is obscure and pregnant with conflicting meanings.”

Of course, Pearl Harbor was an unexpected disaster that seemingly came out of nowhere. While we have those occasionally in business, more often than not our “disasters” come from strategies, redesigns or promotions that did not perform as expected. And those expectations can also lead to our blindness.

Whenever we’re implementing some new and exciting strategy, we tend to be very optimistic about the results. We’re convinced these new strategies are going to provide positive returns or we wouldn’t be implementing them. That optimism can lead to the same sort of crystal clear signal Wohlstetter referenced, but in the opposite direction; i.e. we tend to only see how everything we’re doing will lead to greatness and can easily overlook variables that have potential to lead to negative outcomes.

So, what do we do about it?

It seems some of the most common solutions today involve pulling together a committee to review what went wrong and putting together processes to prevent those specific problems in the future. These new processes don’t prevent all potential problems in the future, but with any luck they’ll prevent us from repeating the same mistakes.

But all of that happens after the fact.

There’s got to be a better way. My problem with the “committee and new process” approach is there’s a tendency to introduce lots of new and –all too often — needless bureaucracy. Inefficiencies ensue without greatly decreasing the probability of problem-free future efforts.

A technique I’ve found effective invokes much of the clarity of hindsight by drawing on the power of imagination.

During the ROI process for the strategy or project, we’ve already imagined the positive outcome. So before we wrap up planning, let’s also imagine a couple horrific scenarios. For example, imagine that four or five months after a site redesign, sales are down 50% and customer satisfaction has tanked. What happened? Now let’s assemble the same type of committee we would in that scenario and pour over the plan to find the causes of our imagined disaster.

Some might say this technique is really just standard contingency planning, but I find some pretty big differences. Contingency planning tends to look at the current plan to identify execution risks. It doesn’t often uncover key strategic or design problems.

The Scenario Imagination technique provides us with a different sort of lens that taps into our hindsight abilities to separate the signal from the noise.

We certainly won’t find every potential problem, but every problem we mitigate increases our probability of success and reduces our risk. And if we can reduce a lot of risk without strangling ourselves in bureaucracy, we’ll likely lower costs, increase efficiencies, and increase profits. I like the sound of that.

What do you think? Have you run into these types of issues? Do you think this technique would work for you? Do you have any techniques you would like to share?

Photo credit: me’nthedogs

Retail: Shaken Not Stirred by Kevin Ertell


Home | About