Are you measuring the right thing with your website optimization tests?
One of the biggest mistakes you can make with A/B or multivariate testing is to have conversion rate myopia. It can cause you to draw the wrong conclusions about success, and apply the wrong elements to your site which can actually cause you to
While it’s always important to measure conversion rate improvement with any test, it’s not the only (and not always the most important) metric to consider in testing.
What does conversion rate mean, anyway?
There are different data points you can use to construct a conversion rate, but essentially it is expressed as the number of conversion actions (sales, completed forms, Facebook Likes, email sign ups, etc) divided by the number of visitors. Where conversion rate can become convoluted is when you have to decide whether a conversion rate applies to total visitors or unique visitors, or to a particular traffic segment (e.g. exclude customers outside of your shipping area).
Know a Test Page’s Goal
For most folks, the conversion goal measured is sales. So a home page test would be tied to ultimate sales. But is it fair to hold, for example, a banner on a home page responsible for higher or lower conversion percentages? The home page is simply one page in a long conversion funnel. Unless you can purchase directly from the home page, it does not influence ultimate conversion. What a home page
do is entice the visitor to stay on your site for more than one page view. This means a successful home page test reduces bounce rate and wins a higher percentage of clicks deeper into your site.
Some sites’ conversion goals are a completed “contact us” form, for example. In cases where there is a very short conversion funnel, home pages may have more influence on the holy-grail-conversion for your business.
Understand the Relationship Between Metrics
It’s always important to consider the revenue improvement, not just conversion rate improvement. Removing product recommendations from shopping cart pages, for example, may improve conversion by 20%, but if average order value fell by 30%, it’s not a winner.
Other examples of analytics relationships:
Price and promotions test:
More people bought, but at a lower price – profit did not increase
More people bought, but items per sale were lower – profit did not increase
Email price, promotions or coupon test:
More people purchased, but a percentage would have purchased anyway without the discount
Pre-checked email opt in test:
More people signed up for email, but reported your messages as spam because they signed up unwittingly
Cart button test:
More people initiated checkout, but abandonment the same because the real problem lies in the funnel
Remove negative reviews test:
More people purchased the item, but more tried to refund because item wasn’t explained truthfully, negative reviews suppressed
Banner ad test:
More traffic was driven to your site, conversion rate decreased, revenue increased
The Other Other Metrics
The cure for conversion rate myopia is to embrace the other important metrics, like:
Revenue per visitor
Average order value
Items per order
Margin per visit (profit)
Margin per customer
Revenue less returns
Lifetime customer value
The challenge is, some of these gold nuggets of data are not available from your analytics tool out-of-the-box with Google Analytics. (Some tools like Yahoo Web Analytics and Omniture allow you to import COGS - cost of goods sold - for example). You may need to integrate your data sources together to get a clearer picture. At the very minimum, you should be looking at revenue, revenue per visitor and average order value.
Originally published Jun 6, 2011 7:30:00 PM, updated October 20 2016