A/B testing is one of those techniques that, if you have enough volume to give you significant results, is pretty much guaranteed to generate better results from your marketing.
Email marketers have known this for ages, but what drives me nuts is that they waste their time on tiny little tests -- instead of tackling some of the bigger, more exciting tests that yield real insights and improvements.
In fact, MarketingSherpa's email survey found that subject lines are still the most commonly tested element in email marketing. Meaning that those few words that get your subscribers to open your emails and see your wonderful offers are what marketers focus on most in their attempts to optimize their email marketing.
While I'm sure this strategy can end up getting you the most tested, optimized subject line that will ever reach an inbox, the impact of these tests are minimal compared to all the other things an email marketer could be testing.
So, are you ready to run some big, exciting tests?
In this blog post, we'll highlight the top email testing tools to try and what to test on them.
What is A/B Testing in Email Marketing?
A/B testing is when you launch and test how engaging two designs are. When you send an A/B tested email, the software you're using quickly identifies which is most engaging to the earliest recipients and sends that format to the rest of your subscriber list.
Email Marketing Tests
1. Test different types of offers in your messages.
Possibly the biggest lever you have in your email marketing is not the few words you use to describe your offer, but rather, the offer itself.
Whether you're testing two ebooks against each other, or an ebook versus a webinar, this test is bound to get you better results overall. The reason this is particularly important is, while you may think your offer is the best thing since the iPod, you may also be wrong.
We started doing this sort of testing religiously back in the summer of 2010 and saw dramatic results. Instead of taking our email list and sending them all our latest ebook, we would take a smaller portion of the list, split it in half, send them each two different offers, and then send the better performing offer to the (larger) remainder of the list. This testing alone increased our monthly email leads 4-8x instantly.
Here are some more specific offer elements you can consider testing:
- Topic: Do certain offer topics resonate better with your audience? For example, we might test one of our ebooks on Facebook against one of our ebooks on Twitter.
- Format: Which offer format does your list prefer? Do they love webinars? How does that compare to their interest in ebooks, kits, free trials, etc.?
- Length/Size: Does your audience prefer smaller, bite-sized offers like tip sheets, or are they hungry for more, like an 80-page ebook? Try testing longer forms of content vs. shorter offers, or one offer vs. a set of offers.
- Name of Offer: Sometimes the way you position your offer can make a difference with your audience. Think ebook vs. guide vs. white paper, or fact book vs. slideshow vs. download.
2. Analyze the landing page you'll be linking.
The goal of your email is not just to get someone to open or click through; it's also to take some action. For example, to download your offer. So don't think of your email in a vacuum. Think of it in the context of driving that particular action, which means optimizing where the action takes place: the landing page.
After all, if you create this great email that drives lots of clicks to your website but then you lose those potential leads at the last stage, it's like you've run the first leg of a marathon but then decided to drop out of the race during the very last mile.
Here are some important landing page elements to test:
- Description of Offer: The way you position your offer may have an impact. Calling out that a consultation is free, or referencing testimonials of people who have downloaded that offer, for example, can be interesting variables to test.
- Length of Description: Do you go on and on about your offer, providing testimonials and screenshots, or do you keep things short and sweet in bullet point form?
- Image/Preview of Offer: Using a supporting image is great, but what do you show? An image of the ebook cover, a sample page of the ebook so people can see what's inside, or a preview of the first few pages?
- Form Placement: Do you put the form on the left? The right? Below a block of text? Best practices say make it visible on immediate page load (above the fold), but feel free to play around with the placement.
- Number of Form Fields: What data do you really need from your prospects? Fewer form fields usually leads to a higher conversion rate, but you should always test asking the bare minimum versus asking for every personal detail -- and somewhere in between. We've also published some great advice about this debate here.
- Which Form Questions to Ask: In addition to the number of form fields, which questions you ask on your form can have a big impact. Asking for Social Security Numbers or visitors' first born child's name is very different from asking for size of company or industry.
- Form "Submit" Button Text: Do you use a straightforward, action-oriented phrase like "Download Ebook Now," a fun option like "Let's Go!" or a standard "Download" button? Test out the text of the button you know each lead is clicking on.
3. Leverage audience segmentation tests.
The success of your email is not just dependent on what you're emailing or how you're emailing it, but also *who* you're emailing.
For HubSpot, an offer called, Agency Kit: How to Create Effective Ebooks for your Clients may get a great response from marketing agency owners, but it'd probably get a terrible response from the nonprofit marketers interested in our content.
The simple act of segmenting your email list to narrow your audience down to one that would find your content more relevant can have an amazing impact on your results.
Here are some audience segmentation tests you can run:
- Interest: Has someone downloaded an ebook on this topic before? Do you know they have a particular challenge based on their website browsing history? Target the offers around those interests for a boost in response rate.
- Persona:Identify your main business personas, and target your content to each one. At HubSpot, this means we send different content to small business owners than what we send to nonprofit marketers, for example.
- Recency or Level of Engagement: Did this subscriber come to your site recently, or has it been a few months? Did they download a dozen ebooks, or just one?
- Other Demographics: Try segmenting on other demographics collected by marketing or sales - things like industry or role or company size.
- Lifecycle Stage: Where is this person in the sales and marketing funnel? Did they just start engaging with you, or are they in the last stages of the sales process? This article provides suggestions on what to send at each stage of the funnel.
Check out this blog post for even more examples of how you can slice and dice your email list for better segmentation.
4. Test different newsletter formats.
Changing up the format of your email can also have a surprising effect on your response rate. This could mean everything from the length of the email, to including a lot of images, to creating a simple, plain text email. Keep in mind that your results may differ depending on the type of offer.
For example, our new ebooks perform best when sent in a nicely formatted html email, while our free consultation offers perform better when sent as a simple, plain text email.
Here are some formatting elements you can test in your email marketing:
- Plain Text vs. HTML: Simply try changing your pretty HTML email into a plain, personal-looking email to see how that changes your response rates. You might be surprised at the results!
- Content in Text Only vs. Text and Images: At HubSpot, for example, we tend not to rely too much on images because many subscribers don't enable or download images in their emails. That being said, some companies have had great success with using visuals to tell stories that you simply can't convey through words alone.
- Number of Calls-to-Action: Do you go with a newsletter style with a lot of calls-to-action, or zone in on one single offer?
- Length of Email: Do you go short and sweet, include meaty content, or go on and on about the value of the offer?
If you have a number of different email templates or design tweaks you want to test in a limited amount of time, you could consider A/B testing.
5. Send newsletters at different times and frequencies.
Timing is one of the most popular things marketers try to optimize. But it seems like there's more talk about the best time to send in general, and not enough testing going on to determine the best time to send email to your own subscribers -- or even a specific segment of your subscribers.
Even within HubSpot, we have segments of subscribers who respond more to emails on Mondays, Saturdays, mornings, afternoons -- on top of that, all in their own timezones.
Instead of sending email at every marketer's favorite time (Tuesdays at 10 a.m.), break away from the pack and see what works specifically for your audience in order to optimize for your particular business -- and to have a better chance of breaking through the clutter of other businesses' emails.
Consider conducting the following timing/frequency tests in your email marketing:
- Day of the Week: If you always email on Tuesdays, try mixing it up and sending on a Monday or Saturday.
- Time of Day: Do you always send emails in the mornings on the East Coast? Try an afternoon send -- or even go for after work hours.
- Triggered by Specific Behavior: It's not just about when you want to send an email, it's about when your subscriber has taken some interesting action. Try targeting your follow-up around when they take an action using marketing automation.
- Timing Around Trigger Event: How soon after the triggering event should you send that email? Immediately? An hour later? A day later? Longer?
- Frequency: How much should you email someone, and how much time should you leave in between? Once a month, once a week, once a day? Check out this article to help you determine your optimal email frequency.
6. Determine if your sender name or address impacts your email numbers.
If you haven't tested a different sender name or address yet, definitely add this to your list. While best practices still apply (in other words, using a name that recipients will recognize as well as a real email address that your prospects can respond to), you can always try out different names to see how it affects your open and clickthrough rates.
Here are some sender name tests to try out:
- Consistency vs. Change: Should you use the same name for consistency, or try changing it up email to email to garner more attention?
- Personal vs. Company: Should you use an individual's name, your company name, or some combination? (e.g. 'Ellie Mirman,' 'HubSpot,' or 'Ellie Mirman, HubSpot')
- Category-Related Name: If you have a subscriber in a particular segment of your business, you can try sending an email from the name of that segment (e.g. 'Small Business Team'). If your subscriber signed up for a particular type of content, try using a name related to that specific content type (e.g. 'HubSpot Webinars)'.
Email Testing Tools
- HubSpot's Free Email Tool
- Litmus
- Mail-Tester
- SenderScore
- SubjectLine
1. HubSpot's Free Email Software
HubSpot's free email tool allows you to create email campaigns that can be ent to email subscribers or contacts in your CRM.
Aside from providing an easy-to-use drag-and-drop software, HubSpot also allows you to test email designs through an A/B test feature. The software also provides tips related to length in the subject line and preview text area which can help you write captions and instantly verify that they won't get cut off.
On top of all these features, HubSpot's software will notify you if it can't find a link in your email. It will also warn you when inboxes like Gmail will trim your message.
2. Litmus
Ever wonder if your email will look aesthetically pleasing on different devices and in different email provider inboxes? With Litmus, you can sign up for free and send them the email you want to test. From there, Litmus will automatically review the email and send you screenshots showing what your message will look like to readers using different email providers.
3. Mail-Tester
Worried you're using phrases or wording that could trigger spam filters to burry your email? With Mail-Tester, you can log in and get a special email address to send your test email to. After you send it, Mail-Tester will send you a report that notes any of the spammy trigger words that were in your message so you can correct your language before sending to your full list.
4. Sender Score
Sometimes, if your IP address is associated with sending many different email newsletters, email providers might move your email to spam. If you suspect that your IP address could be negatively impacting your email numbers, you can use SenderScore to find out if your IP address is considered "spammy."
5. SubjectLine
If you need help with writing subject lines, you can test a few before sending your email with SubjectLine.com. When you go to the website, you simply type in a subject line and click the submit button. Then, you'll receive a grade out of 100 points as well as pointers for improvement.
Reminder: Test only one thing at a time.
The key with any of these tests is to test just one element at a time so you can isolate your variables and thus tie the difference in results is to that particular change. And if you crank through this list of BIG email tests, here are some great ideas for quick, smaller tests.
Editor's note: This post was originally published in December 2012 and has been updated for comprehensiveness.
![How to Understand & Calculate Statistical Significance [Example]](https://www.hubspot.com/hubfs/FEATURED%20IMAGE-Nov-22-2022-06-25-18-4060-PM.png)