Artificial intelligence is all the buzz these days. But how does it actually work? Over the next few weeks, we'll take you behind the scenes of HubSpot's artificial intelligence, one feature at a time. Our first post covered duplicate management. Today, we’ll walk you through adaptive testing — the next evolution of A/B testing, infused with AI.
If you’ve been around the marketing world for more than a few days, I bet you’ve run an A/B test. The idea of an A/B test is simple: to achieve a specific conversion goal, you publish two versions of a marketing asset instead of one. Viewers of the asset see one of the variations at random. After a certain amount of time, you check the performance of the two variations and pick a winner. From that point forward, everyone sees the winning version.
A/B testing is powerful. Let's look at one example: using A/B testing, then-candidate Barack Obama increased conversion on his home page to the tune of $60 million in donations by testing variations of two simple elements — the banner image and the button text.
The Problem: How A/B Testing is Broken
While it is certainly powerful, A/B testing is fundamentally flawed in two specific ways:
First, the process of choosing a winner is manual. That’s both time-intensive (you need to check on results regularly) and intellectually challenging. Let’s be honest: despite a dozen Google searches over the years, how many of us could actually define statistical significance?
Second, half the visitors see the worse variation until you pick a winner. In the Obama example above, let’s say Dan (Obama's chief marketer) was pitting a red “sign up” button against a blue one. And let’s say that, right away, red started to perform better. Until he made a manual change — likely after waiting for a critical mass of traffic to the page — the worse-performing blue button would continue to receive half the traffic to the page. The results would look like this:
The Solution: Adaptive Testing
We brought this problem to our AI engineers and they had the perfect solution: multi-armed bandit testing. We thought they were kidding. We asked for an application of AI, not an idea for next year’s Halloween costume. As it turns out, they were completely serious.
The multi-armed bandit problem is well-known in the world of artificial intelligence. Here's the idea: it models a system that’s simultaneously trying to find new knowledge (“exploration”) and optimize its decisions based on existing knowledge (“exploitation”). In other words, it’s an ideal way to evolve A/B testing: you’re looking to test new variations of assets while also optimizing the variations you show to users based on past performance.
The idea behind multi-armed bandit testing was perfect. The name…not so much. So we called it “adaptive testing” instead.
Here’s how adaptive testing works in the context of your HubSpot account.
Let’s say you test four variations of a page. With adaptive testing, traffic is split evenly between page variations at first. As HubSpot learns how these variations are performing, we adjust the traffic automatically, so that better-performing variations are shown more and poorer-performing variations are shown less. The traffic distribution of an adaptive test might look like this:
As you can see, the better variation wins out in the end. There are a few advantages of using adaptive testing over traditional A/B testing:
- Since you, as the experimenter, don’t have to wait to see how variations are performing before picking a winner, adaptive testing delivers results (i.e. improvements in form submission rates) much faster. Plus, no need to manually check results, so you can leave that statistical significance calculator in the drawer.
- Because the distribution is optimized over time, you improve conversion rates during the testing period. In adaptive tests, the better-performing variation starts to win out right away, whereas in classic A/B testing you show 50% of visitors each variation.
P.S. Want the full details of the models that make adaptive testing work? Here’s a post written by one of our engineers that gets into the weeds.
How It Works
Want to run an adaptive test in your HubSpot account? Here’s how to get started.
1. Run a test
Start in HubSpot's CMS page editor and click “run a test.” Choose “adaptive test.”
2. Select a goal and number of variations
Next, choose your goal and add up to five variations. Give each one a unique name.
3. Perfect your variations
Once you’ve created your variations, the “run a test” button gets replaced with a dropdown that allows you to toggle between your variations and edit them to your liking.
4. Publish
When you’re done editing all your variations, click “publish.” Easy as that!
5. Analyze your test
Once your test has run for a sufficient period of time, data from all page variations will appear in this “test results” tab. There are three primary charts to help you understand your results.
- Determine which variation best accomplished your goal using the “goal metric comparison” graph. In this example, the “beach” variation performed best.
- Understand how the decision was made using the “traffic distribution over time” graph. This graph shows how traffic was allocated to the different variations over time. In this example, at first, the variations were distributed evenly. Then, as the model learned that the “beach” variation was performing best, it allocated more traffic there. Eventually, the variation won out and received all traffic.
- Deep dive into the data in the comprehensive table, which shows all metrics for every variation in the test.
6. End your test
The best part about adaptive tests is that they’re “set it and forget it.” No need to officially end a test; it runs forever, optimizing performance over time. If you don’t like the idea of a never-ending test, you can end the test yourself by clicking “end test” and choosing your winning variation.
And that’s it! You’ve created four variations of your page and let HubSpot take care of the rest.
Three Examples to Get You Started
How might you actually implement adaptive testing? Here are some ideas to get you off the ground.
- Test different videos on a page. Here's a real HubSpot example: a German fitness center is testing two videos on their page to see which leads to more memberships.
- Test background images. One HubSpot customer who makes tax software is testing different page variations to figure out which background style leads to the most eBook downloads.
- Test your layouts. One publisher that uses HubSpot is testing multiple landing page layouts to determine which leads to more consultation bookings.
Ready to start optimizing? Adaptive testing is now available in all Marketing Hub Enterprise accounts. Start using it now.
Want to connect with others on HubSpot tips, tricks, and updates? Head over to the HubSpot Community to join a conversation or start one of your own.
