Posts tagged: assumptions

Blinded By Certainty

blindfoldedIn reality, very little in our lives is absolutely certain. We can be certain the sun will rise in the east and set in the west. We can be certain death will follow life. And we can be pretty darn certain Steve Jobs will wear a black turtleneck and jeans at his next public appearance.

But we’re certain about a lot more things than we should be.

A recent University of Michigan study by Brendan Nylan and Jason Reifler shows that the more certain we are about particular ideas or situations the more we become blind to facts that discredit our certainty. In fact, in many cases opposing facts are not just ignored but actually strengthen our prior beliefs.  A recent Boston Globe article provides an excellent summary of the research.

From the article:

Most of us like to believe that our opinions have been formed over time by careful, rational consideration of facts and ideas, and that the decisions based on those opinions, therefore, have the ring of soundness and intelligence. In reality, we often base our opinions on our beliefs, which can have an uneasy relationship with facts. And rather than facts driving beliefs, our beliefs can dictate the facts we chose to accept. They can cause us to twist facts so they fit better with our preconceived notions. Worst of all, they can lead us to uncritically accept bad information just because it reinforces our beliefs. This reinforcement makes us more confident we’re right, and even less likely to listen to any new information.

Both the research and the article focus primarily on our political viewpoints, but while reading I couldn’t help but think of people I’ve come across in the business world who were unbelievably certain about their viewpoints based on information or experiences that seemed less than obvious to me. I immediately thought of dozens of people, and I bet you’re thinking of many such people now.

In fact, it was so easy for me to think of other people that fit the bill that I couldn’t help but think the man in the mirror was not immune to this universal human fallacy.

In my experience in the business world, we often assume with undue certainty that past experiences will reflect future possibilities. We say things like, “We tried that before and it didn’t work” or “I know what our customers want.” While our past experiences are extremely valuable and are very important for informing future decisions, we simply don’t have enough of them to blindly ignore changes in circumstances, timing and other variables that could significantly alter results for a new effort.

So how do we overcome our natural instincts in order to make better business decisions?

  1. Be aware of the problems with certainty
    You’ve read this far, so maybe you’re awareness is already active. I know that I am reassessing all the things I “know” to try to truly separate what is fact and what is assumption. I very much value all my experience, and I know I make better decisions because of what I’ve seen and heard along the way. But I want to make doubly sure that assumptions I make based on past experiences are tested and validated before I turn them into absolute fact.
  2. Actively seek alternate points-of-view
    In my experience, the combination of multiple experiences provides a much more solid foundation for decision making than basing decisions on singular past experiences. Techniques I’ve used, like The Monkey Cage Sessions, are based on the incorporating viewpoints from people in different functional areas and levels of the organization. While it’s acceptable to discount data or opinions that are in opposition to a decision I might make, I want to be sure I’m not simply rationalizing opposing information or viewpoints solely because they are different from my biases.
  3. Envision alternate scenarios
    I addressed this some in a previous post, “Obscure and pregnant with conflicting meanings”, where I discussed a technique I called “Scenario Imagination.” I’ve since read an excellent interview with Daniel Kahneman and Gary Klein where they detail a similar and better technique they call “pre-mortem” (which is also a better name than mine). Whenever we make decisions, we have a tendency to assume our decisions are going to produce the best possible results. These pre-mortem techniques have us imagine worst case scenarios to try to dissect potential problems before they occur.
  4. Be flexible and plan for contingencies
    Once we admit we’re not 100% certain, we can move forward with plans that are flexible and able to react to changing conditions. To be clear, I’m not saying we should just be wishy-washy and not make clear decisions. What I’m saying is that we should be open to new facts and be sure we have created an environment that allows us to change course when warranted.

If we’re aware of our certainty biases and take active steps to address them, I believe we can significantly improve our decision-making in our businesses.

What do you think? Upon self-examination, have you turned beliefs into facts in your mind? How would you suggest addressing these biases? Or, do you think is all a load of hooey?

11 Ways Humans Kill Good Analysis

Failure to CommunicateIn my last post, I talked about the immense value of FAME in analysis (Focused, Actionable, Manageable and Enlightening). Some of the comments on the post and many of the email conversations I had regarding the post sparked some great discussions about the difficulties in achieving FAME. Initially, the focus of those discussions centered on the roles executives, managers and other decisions makers play in the final quality of the analysis, and I was originally planning to dedicate this post to ideas decision makers can use to improve the quality of the analyses they get.

But the more I thought about it, the more I realized that many of the reasons we aren’t happy with the results of the analyses come down to fundamental disconnects in human relations between all parties involved.

Groups of people with disparate backgrounds, training and experiences gather in a room to “review the numbers.” We each bring our own sets of assumptions, biases and expectations, and we generally fail to establish common sets of understanding before digging in. It’s the type of Communication Illusion I’ve written about previously. And that failure to communicate tends to kill a lot of good analyses.

Establishing common understanding around a few key areas of focus can go a long way towards facilitating better communication around analyses and consequently developing better plans of action to address the findings.

Here’s a list of 11 key ways to stop killing good analyses:

  1. Begin in the beginning. Hire analysts not reporters.
    This isn’t a slam on reporters, it’s just recognition that the mindset and skill set needed for gathering and reporting on data is different from the mindset and skill set required for analyzing that data and turning it into valuable business insight. To be sure, there are people who can do both. But it’s a mistake to assume these skill sets can always be found in the same person. Reporters need strong left-brain orientation and analysts need more of a balance between the “just the facts” left brain and the more creative right brain. Reporters ensure the data is complete and of high quality; analysts creatively examine loads of data to extract valuable insight. Finding someone with the right skill sets might cost more in payroll dollars, but my experience says they’re worth every penny in the value they bring to the organization.
  2. Don’t turn analysts into reporters.
    This one happens all too often. We hire brilliant analysts and then ask them to spend all of their time pulling and formatting reports so that we can do our own analysis. Everyone’s time is misused at best and wasted at worst. I think this type of thing is a result of the miscommunication as much as a cause of it. When we get an analysis we’re unhappy with, we “solve” the problem by just doing it ourselves rather than use those moments as opportunities to get on the same page with each other. Web Analytics Demystified‘s Eric Peterson is always saying analytics is an art as much as it is a science, and that can mean there are multiple ways to get to findings. Talking about what’s effective and what’s not is critical to our ultimate success. Getting to great analysis is definitely an iterative process.
  3. Don’t expect perfection; get comfortable with some ambiguity
    When we decide to be “data-driven,” we seem to assume that the data is going to provide perfect answers to our most difficult problems. But perfect data is about as common as perfect people. And the chances of getting perfect data decrease as the volume of data increases. We remember from our statistics classes that larger sample sizes mean more accurate statistics, but “more accurate” and “perfect” are not the same (and more about statistics later in this list). My friend Tim Wilson recently posted an excellent article on why data doesn’t match and why we shouldn’t be concerned. I highly recommend a quick read. The reality is we don’t need perfect data to produce highly valuable insight, but an expectation of perfection will quickly derail excellent analysis. To be clear, though, this doesn’t mean we shouldn’t try as hard as we can to use great tools, excellent methodologies and proper data cleansing to ensure we are working from high quality data sets. We just shouldn’t blow off an entire analysis because there is some ambiguity in the results. Unrealistic expectations are killers.
  4. Be extremely clear about assumptions and objectives. Don’t leave things unspoken.
    Mismatched assumptions are at the heart of most miscommunications regarding just about anything, but they can be a killer in many analyses. Per item #3, we need to start with the assumption that the data won’t be perfect. But then we need to be really clear with all involved what we’re assuming we’re going to learn and what we’re trying to do with those learnings. It’s extremely important that the analysts are well aware of the business goals and objectives, and they need to be very clearly about why they’re being asked for the analysis and what’s going to be done with it. It’s also extremely important that the decision makers are aware of the capabilities of the tools and the quality of the data so they know if their expectations are realistic.
  5. Resist numbers for number’s sake
    Man, we love our numbers in retail. If it’s trackable, we want to know about it. And on the web, just about everything is trackable. But I’ll argue that too much data is actually worse than no data at all. We can’t manage what we don’t measure, but we also can’t manage everything that is measurable. We need to determine which metrics are truly making a difference in our businesses (which is no small task) and then focus ourselves and our teams relentlessly on understanding and driving those metrics. Our analyses should always focus around those key measures of our businesses and not simply report hundreds (or thousands) of different numbers in the hopes that somehow they’ll all tie together into some sort of magic bullet.
  6. Resist simplicity for simplicity’s sake
    Why do we seem to be on an endless quest to measure our businesses in the simplest possible manner? Don’t get me wrong. I understand the appeal of simplicity, especially when you have to communicate up the corporate ladder. While the allure of a simple metric is strong, I fear overly simplified metrics are not useful. Our businesses are complex. Our websites are complex. Our customers are complex. The combination of the three is incredibly complex. If we create a metric that’s easy to calculate but not reliable, we run the risk of endless amounts of analysis trying to manage to a metric that doesn’t actually have a cause-and-effect relationship with our financial success. Great metrics might require more complicated analyses, but accurate, actionable information is worth a bit of complexity. And quality metrics based on complex analyses can still be expressed simply.
  7. Get comfortable with probabilities and ranges
    When we’re dealing with future uncertainties like forecasts or ROI calculations, we are kidding ourselves when we settle on specific numbers. Yet we do it all the time. One of my favorite books last year was called “Why Can’t You Just Give Me the Number?” The author, Patrick Leach, wrote the book specifically for executives who consistently ask that question. I highly recommend a read. Analysts and decision makers alike need to understand the of pros and cons of averages and using them in particular situations, particularly when stacking them on top of each other. Just the first chapter of the book Flaw of Averages does an excellent job explaining the general problems.
  8. Be multilingual
    Decision makers should brush up on basic statistics. I don’t think it’s necessary to re-learn all the formulas, but it’s definitely important to remember all the nuances of statistics. As time has passed from our initial statistics classes, we tend to forget about properly selected samples, standard deviations and such, and we just remember that you can believe the numbers. But we can’t just believe any old number. All those intricacies matter. Numbers don’t lie, but people lie, misuse and misread numbers on a regular basis. A basic understanding of statistics can not only help mitigate those concerns, but on a more positive note it can also help decision makers and analysts get to the truth more quickly.

    Analysts should learn the language of the business and work hard to better understand the nuances of the businesses of the decision makers. It’s important to understand the daily pressures decision makers face to ensure the analysis is truly of value. It’s also important to understand the language of each decision maker to shortcut understanding of the analysis by presenting it in terms immediately identifiable to the audience. This sounds obvious, I suppose, but I’ve heard way too many analyses that are presented in “analyst-speak” and go right over the heard of the audience.

  9. Faster is not necessarily better
    We have tons of data in real time, so the temptation is to start getting a read almost immediately on any new strategic implementation, promotion, etc. Resist the temptation! I wrote a post a while back comparing this type of real time analysis to some of the silliness that occurs on 24-hour news networks. Getting results back quickly is good, but not at the expense of accuracy. We have to strike the right balance to ensure we don’t spin our wheels in the wrong direction by reacting to very incomplete data.
  10. Don’t ignore the gut
    Some people will probably vehemently disagree with me on this one, but when an experienced person says something in his or her gut says something is wrong with the data, we shouldn’t ignore it. As we stated in #3, the data we’re working from is not perfect so “gut checks” are not completely out of order. Our unconscious or hidden brains are more powerful and more correct than we often give them credit for. Many of our past learnings remain lurking in our brains and tend to surface as emotions and gut reactions. They’re not always right, for sure, but that doesn’t mean they should be ignored. If someone’s gut says something is wrong, we should at the very least take another honest look at the results. We might be very happy we did.
  11. Presentation matters a lot.
    Last but certainly not least, how the analysis is presented can make or break its success. Everything from how slides are laid out to how we walk through the findings matter. It’s critically important to remember that analysts are WAY closer to the data than everyone else. The audience needs to be carefully walked through the analysis, and analysts should show their work (like math proofs in school). It’s all about persuading the audience and proving a case and every point prior to this one comes into play.

The wealth and complexity of data we have to run our businesses is often a luxury and sometimes a curse. In the end, the data doesn’t make our businesses decisions. People do. And we have to acknowledge and overcome some of our basic human interaction issues in order to fully leverage the value of our masses of data to make the right data-driven decisions for our businesses.

What do you think? Where do you differ? What else can we do?

The Tree Stump Theory

Since I mentioned it in my eTail presentation last week, I’ve received a number of requests to expound on my Tree Stump Theory in this space. So, here goes:

As truly amazing as the human brain is, it’s not able to re-process everything we see anew every time we see it. So, our brains take some shortcuts by basically ignoring things we are very familiar with, and that can cause us trouble any time we have interactions with people who don’t have the same level of familiarity with something as we do. I usually talk about this in reference to website usability, but it actually applies to many areas of our lives. To illustrate the concept, I have my Tree Stump Theory…

Imagine if someone brought a big tree stump into one of your conference rooms. The first time you saw it, you would say something like “Hey, what’s with the tree stump?” Someone would give you a compelling reason why it was there, and you would go on with the meeting. The next time you entered the conference room, you would notice the tree stump but not ask about it. After while, someone might throw a tablecloth on it or dress it up in some manner, but it would still be there. You would no longer ask about it or think about it. Frankly, you wouldn’t even really see it. You’d just arrange yourselves at the table in a way that worked around the tree stump and go on with your meeting. Meanwhile, anyone new coming into the room can’t help but see the tree stump and find it to be an obstacle.

We all have these types of “tree stumps” on our sites and in our lives. I bet you could think of something like this in your house right now. They manifest themselves as obstacles to good web usability, but they’re also our biases, our stereotypes and any other set of assumptions we rely on, usually unconsciously, to drive our daily actions and decisions. Sometimes they’re relatively harmless, but more often than not tree stumps prevent people from buying on our sites, or they are the unspoken roots of disagreements and miscommunications in our daily interactions both at work and at home.

So how do we get rid of our tree stumps?

1. The first step is to recognize the fact that tree stumps are everywhere, even when we can’t see them.

If you’ve read this far, you’ve probably made it to step one.

2. Next, get some help finding them

The very nature of tree stumps makes them difficult to self-identify. If you’re dealing with web usability, try the steps prescribed in this post. If you’re concerned about tree stumps in strategies, policies or general decisions, seek some input from someone who is outside the general team and who has a different background from you and your key decision makers. Ask them to openly question everything.

3. Specifically call out assumptions, preferably in writing

Assumptions are the roots of tree stumps. We make assumptions so often that we don’t always realize we’re making them. Listen for statements or reasons that hint of tree stumps. The most obvious is “That’s the way we’ve always done it.” If you hear that one, sound the sirens. But there are other, less obvious comments like “People want…” or “Based on my experience…” or “In a previous life we…” Don’t get me wrong, some of these statements could be perfectly accurate and valid. But whenever someone is applying past experience to a currently situation, he or she is assuming the two situations are similar enough to warrant the comparison. That’s potentially an assumption fraught with problems because the number of potentially important variables in any situation is massive. Writing down those assumptions and then testing them on the current situation often brings bad assumptions to light.

Also, on the web usability side, remember that while your internal reason for a tree stump may seem extremely valid to everyone in the company, your customers don’t know those reasons and even if they did, they probably don’t care. Common explanations that won’t hold water with customers include:”I’m not in charge of that area;” “It doesn’t matter because people don’t use that anyway;” or the time-honored classic, “That’s due to the limitations of our platform.”

4. Schedule regular reviews of your own assumptions

This one in some ways is a repeat of #3, but the point here is to specifically and methodically question yourself. This is really hard to do, of course, but it has a tremendous amount of value. One technique I’ve used in various situations is to write down my first impressions of important situations so that I can regularly review them in the future after I’ve learned more. I recently talked with about this technique in reference to starting a new job. Beyond that technique, it just takes practice and discipline to think about your own biases and assumptions to see if they still apply.

I also find it helpful to constantly look for new ideas. I read lots of business and science books. I don’t always agree with everything I read, but new ideas cause me to question my own ideas. I also enjoying reading thought-provoking blogs, some of which are listed to the right, and I follow interesting people on Twitter. More than anything, though, I love to spend time talking to people who think differently than I do and are willing to share their perspectives. (And I hope you’ll share your comments on this post and others.)

Tree stumps are everywhere. We’ve all got them. And as soon as we remove some, more will crop up. It takes a concerted effort and a solid process to regularly look for and remove the tree stumps in our lives and our businesses. But I’ll argue that those of us who are aware of our tree stumps are on a much faster path to improvement than those who go on ignoring them.

What do you think? What types of tree stumps have you run into? How do you go about removing them?

Retail: Shaken Not Stirred by Kevin Ertell

Home | About