I’ve found it’s best to break our metrics down to three levels in order to make the most of them.

1. KPIs
The first and highest level of metrics contains the Key Performance Indicators or KPIs. I believe strongly there should be relatively few KPIs — maybe five or six at most — and the KPIs should align tightly with the company’s overall business objectives. If an objective is to develop more orders from site visitors, then conversion rate would be the KPI. If another objective is about maximizing the customer experience, then customer satisfaction is the right metric.

In addition to conversion rate and customer satisfaction, a set of KPIs might include metrics like average order value (AOV), market share, number of active customers,  task completion rate or others that appropriately measure the company’s key objectives.

I’ve found the best KPI sets are balanced so that the best way to drive the business forward is to find ways to improve all of the KPIs, which is why businesses often have balanced scorecards. The reality is, we could find ways to drive any one metric at the expense of the others, so finding the right balance is critical. Part of that balance is ensuring that the most important elements of the business are considered, so it’s important to have some measure of employee satisfaction (because employee satisfaction leads to customer satisfaction) and some measure of profitability.  Some people look at a metric like Gross Margin as the profitability measure, but I prefer something deeper down the financial statement like Contribution Margin or EBITDA because they take other cost factors like ad spend, operational efficiencies, etc. into account and can be affected by most people in the organization.

It’s OK for KPIs to be managed at different frequencies. We often talk about metrics dashboards, and a car’s dashboard is the right metaphor. Car manufacturers have limited space to work with, so they include only the gauges the most help the driver operate the car. The speedometer is managed frequently while operating the car. The fuel gauge is critically important, but it’s monitored only occasionally (and more frequently when it’s low). Engine temperature is a hugely important measure for the health of the car, but we don’t need to do much with it until there’s a problem. Business KPIs can be monitored in a similarly varied frequency, so it’s important that we don’t choose them based on their likelihood to change over some specific time period. It’s more important to choose the metrics that most represent the health of the business.

2. Supporting Metrics
I call the next level of metrics Supporting Metrics. Supporting Metrics are tightly aligned with KPIs, but they are more focused on individual functions or even individual people within the organization. A KPI like conversion rate can be broken down by various marketing channels pretty easily, for example. We could have email conversion rate, paid search conversion rate, direct traffic conversion rate, etc. I also like to look at True Conversion Rate, which measures conversion against intent to buy.

Supporting metrics should be an individual person’s or functional area’s scorecard to measure how their work is driving the business forward. Ensuring supporting metrics are tightly aligned with the overall company objectives helps to ensure work efforts throughout the organization are tightly aligned with the overall objectives.

As with KPIs, we want to ensure any person or functional area isn’t burdened with so many supporting metrics that they become unmanageable. And this is an area where we frequently fall down because all those metrics and data points are just so darn alluring.

The key is to recognize the all-important third level of metrics. I call them Forensic Metrics.

3. Forensic Metrics
Forensic Metrics are just what they sound like. They’re those deep-dive metrics we use when we’re trying to solve a problem we’re facing in KPIs or Supporting Metrics. But there are tons of them, and we can’t possibly manage them on a day-to-day basis. In the same way we don’t dust our homes for prints every day when we come home from work, we can’t try to pay attention to forensic metrics all the time. If we come home and find our TV missing, then dusting for prints makes a lot of sense. If we find out conversion rate has dropped suddenly, it’s time to dig into all sorts of forensic metrics like path analysis, entry pages, page views, time on site, exit links, and the list goes on and on.

Site analytics packages, data warehouse and log files are chock full of valuable forensic metrics. But those forensic metrics should not find their way onto daily or weekly managed scorecards. They can only serve to distract us from our primary objectives.

—————————————————–

Breaking down our metrics into these three levels takes some serious discipline. When we decide we’re only going to focus on a relatively small number of metrics, we’re doing ourselves and our businesses a big favor. But it’s really important we’re narrowing that focus on the metrics and objectives that are most driving the business forward. But, heck, we should be doing that anyway.

What do you think? How do you break down your metrics?

In Moneyball, Michael Lewis tells the story of Beane defying the conventional wisdom of longtime baseball scouts and  baseball industry veterans. Rather than trust scouts who would literally  determine a baseball player’s prospects by  how he physically looked, Beane went to the data as a disciple of Bill James’ Sabermetrics theories. By following the  James’ approach, Beane was able to put together consistently winning teams while working with one of the lowest payrolls in the Major Leagues.

Lewis describes how James took a new look at traditional baseball statistics and created new statistics that were  actually more causally related to winning games. Imagine that! For example, James found on-base percentage, which  includes walks when calculating how often a player gets on base, to be a much more reliable statistic than batting  average, which ignores walks (even though we’re always taught as Little Leaguers that a walk is as good as a hit). I won’t get into all the details, but suffice to say on-base percentage is more causally related to scoring runs than batting  average, and scoring runs is what wins games.

So why is batting average still so prevalent and what does this have to do with retail?

Basically, an English statistician named Henry Chadwick developed batting average as a statistic in the late 1800s and didn’t include walks because he thought they were caused by the pitcher and therefore the batter didn’t deserve credit for not swinging at bad pitches. Nevermind that teams with batters who got on base scored more runs and won more games. But batting average has been used so long that we just keep on using it, even when it’s been proven to not be very valuable.

OK, baseball boy, what about the retail?

As relatively young as the e-commerce space is, I believe we are already falling prey to  conventional wisdom in some of our metrics and causing ourselves unnecessary churn.  My favorite example is site conversion rate. Conversion is a metric that has been used in physical retail for a very long time, and it makes good sense in stores where the overwhelming purpose is to sell products to customers on their  current visit.

I’ll argue, though, that our sites have always been about more than the buy button, and they are becoming more and more all-purpose every day. They are marketing and merchandising vehicles, brand builders, customer research tools (customers researching products and us researching customers), and sales drivers, both in-store and online. Given the multitude of purposes of our sites, holding high a metric that covers only one purpose not only wrongly values our sites, but it also causes us to churn unnecessarily when implementing features or marketing programs that encourage higher traffic for valuable purposes to our overall businesses that don’t necessarily result in an online purchase on a particular day.

We still need to track the sales generating capabilities of our sites, but we want to find a causal metric that actually focuses on our ability or inability to convert the portion of our sites’ traffic that came to buy. We used our site for many purposes at Borders, so we found that changes in overall site conversion rate didn’t have much to do at all with changes in sales.

If we wanted to focus on a metric that tracked our selling success, we needed to focus on the type of traffic that likely came with an intent to buy (or at least eliminate the type of traffic that came for other reasons), and we knew through our ForeSee Results surveys that our customers who came with an intent to buy on that visit was only a percentage of our total visitors, while the rest came for other reasons like researching products, finding stores, checking store inventory, viewing video content, etc.

So, how could we isolate our sales conversion metrics to only the traffic that came with an intent to buy?
Our web analyst Steve Weinberg came up with something we called “true conversion” that measured adds to cart  divided by product page views multiplied by orders divided by checkout process starts. This true conversion metric was far more correlative to orders than anything else, so it was the place to initially focus as we tried to determine if we could turn the correlation into causation. We still needed to do more work matching the survey data to path analysis to further refine our metrics, but it was a heckuva lot better than overall site conversion, which was basically worthless to us.

Every site is different, so I don’t know that all sites could take the exact same formula described above and make it work. It will take some work from your web analyst to dig into the data to determine customer intent and the pages that drive your customers ability to consummate that intent. For more ideas, I highly recommend taking a look at Bryan Eisenberg‘s excellent recent topic called How to Optimize Your Conversion Rates where he explores some of these topics in more detail.

—————————————–

Whether or not you buy into everything written in Moneyball or all of Billy Beane’s methods, I believe the main lesson to be culled from the book is that it’s critically important that we constantly re-evaluate our thinking (particularly when conventional wisdom in assumed to be true) in order to get at deeper truths and clearer paths to success.

How is overall site conversion rate working for you? Do you have any better metrics? Where have you run into trouble with conventional wisdom?