How to A/B Test Email Marketing Campaigns for Better Results?

Email marketing is one of the most effective ways of staying connected with your users, that much is clear. But how do you know what users want to see?

One of the best ways to find out is A/B testing. A/B testing helps you understand what your users like to see and more importantly, what they don’t. This includes finding out what type of offers, phrases, buzzwords, and even emojis motivate users to click on your email.

Once you motivate users to click on your email, you can further generate leads, sales, and even engagement.

Email A/B testing — Gmail Window
Photo by Sigmund on Unsplash

In this guide, we’ll walk you through how you can use A/B testing to build a robust email marketing campaign. With that in mind, here’s what you can expect in this guide:

What is A/B testing & Why you need it?

In the world of email marketing, A/B testing (also known as split testing) is when you create and send two variations of the same email campaign to two or more subsets of your audience and based on the responses, decided which one is more effective.

Here we divide the two subsets into a control group and sequential group. The control group is the base group with little to no effect of variations. On the other hand, the sequential group has multiple differences from the base group with little to no control.

Here’s a quick example of what AB testing looks like:

A/B testing example 01-Obama/Biden
Source: https://www.reliablesoft.net/wp-content/uploads/2016/11/Obama-AB-Test.png

What can you test?

A/B testing is useful for testing a number of elements in your email campaign. The most common elements you can test are:

Subject lines

One of the most visible components of your campaign is the subject line. Most devices format subject lines with darker, heavier text so they stand out among other parts of the email.

A/B testing subject length example 02-Topshop
Source: https://www.campaignmonitor.com/wp-content/uploads/2016/02/subjectline.jpg

Given its prominence in your inbox and its influence on open rates, you should pay close attention to subject lines.

Photos/Images

Images help convey what even a thousand words won’t be able to. At the same time, they can also be distracting. If you want users to focus on your design without having to read much, images are a great help. On the other hand, if you want users to focus on your content, images can be distracting.

A little while ago SitePoint experimented with their emails by including an image. But, this showed a decrease in their conversions and click-through rate.

A/B testing image use example 03-SitePoint
Source: https://www.campaignmonitor.com/wp-content/uploads/2016/02/SitePoint-AB-test-email.jpg

A/B testing with images can help you determine if they work for you by examining whether they detract from others or complement them.

Personalization

Users love a personalized message. Studies have shown that personalizing an email campaign can boost click-through rates by up to 14%.

DropBox uses this knowledge to its advantage, including the company’s name of subscribers in emails to make the invitation more relevant to them and increase their likelihood of attending an event.

A/B testing personalization example 04-DropBox
Source: https://www.campaignmonitor.com/wp-content/uploads/2016/02/Company-Personalization-in-Email-Copy.jpg

You can also use your user’s name to catch their eye. When we refer to users with their name, instead of a generic greeting, we’re more likely to build a personal connection.

Choice of words

Words are free, it’s how you use them that defines whether they’ll help you or cost you. This includes the order of words, tone, and even quantity of content used.

If you place the words in your email subject line in the right order, you may be able to influence the way people perceive them and increase your email open rate.

Take a look at these two subject lines for the same email below:

The second variation starts with the fact that opening the email will result in 25% off the next purchase. English-speaking subscribers read from left to right, which highlights the benefits users would get when they open your email.

Similarly, you can drastically influence the results of your email campaigns based on the tone you use in your content.

A study shows that incorporating positivity into your copy engages readers more proactively, helping them to grasp the key messages and driving them to click through and purchase your product.

Call to action (CTA)

Your email marketing campaigns need an effective call to action to be successful. CTAs increase the click-through rate of your emails by clearly communicating to readers what the next steps are.

Sephora uses CTAs brilliantly in their email campaigns. Here’s how.

A/B testing CTA example 05-Sephora
Source: https://www.campaignmonitor.com/wp-content/uploads/2016/02/Sephora-CTa-button-AB-test.jpg

Calls to action are essential for driving click-throughs, so it’s a good idea to A/B test them for optimum results. Here are two things to test in your CTAs:

Layout

There are many numerous types of visual layouts and designs you can use in your email campaigns.

Here are two examples of companies that used an email campaign to announce new features in their products.

A/B testing layout example 06- Freshbooks-SmugMug
Source: https://www.campaignmonitor.com/wp-content/uploads/2016/02/Freshbooks-SmugMug-Visuals.jpg

In comparison, Freshbooks uses a drawing style that closely matches the website’s design, while SmugMug shows a screenshot taken from a Mac laptop.

What works best? This entirely depends on the story you want to portray, your brand, and your target audience.

These are some of the most important features in your email campaign that you need to test with A/B testing. However, there are others as well. Thes include:

How to A/B test your emails?

Now that you know what you’ll be testing, it’s time to dive into how you’ll run A/B tests on your email campaigns.

Step 1. Choose what you want to test

While it’s good to test multiple aspects of your email campaigns, it’s also important to understand you can’t test everything at once. If you want to test everything to design the perfect (or at least the next best thing) email campaign, you’ll have to run A/B tests multiple times.

In this stage, you’ll decide what you want to test. Some options include:

The easiest way to conduct an A/B test is to use email management software such as Unlayer Studio or Mailchimp. For instance, Unlayer Studio allows you to select whether you want to A/B test your subject line or email content.

At this stage, you should ask yourself a few questions to understand which areas are more important for testing. These can be:

In order to drive sales, for example, you might experiment with A/B testing your subject line to encourage more open rates or your call to action (CTA) to encourage users to visit your website and place an order.

Then, create your A/B variations based on the element you decide to test.

Step 2. Determine how you’ll measure your success

Decide how you’ll measure success for your email campaign before launching your email A/B test.

Using what metric will you decide whether version A is better than version B?

If you’re testing the subject line of an e-mail, for instance, you can use your open rate to determine which version will perform better. Similarly, if you’re testing your email’s call-to-action, you can use click-through rates to determine which variation performs better.

Whatever strategy you choose, make sure it’s something you can measure.

Step 3. Choose your audience

Oftentimes, companies have multiple target audiences. Depending on your company and users, you should choose if you want to email your entire list of subscribers or a select few. Once you’ve selected your audience, take two small subsets from that sample for A/B testing.

On average, a successful A/B test usually requires a minimum of 1000 subscribers. It goes without saying that if you have fewer than 1000 subscribers, that doesn’t mean you shouldn’t A/B test. The results of A/B testing can still help you understand your audience better. As you grow your list, you’ll probably see new user behaviors.

Here, it’s also important that you understand where your company stands and where your users stand. In his book, Sun Tzu talked about one of his rules for war which states, “Know thyself, know thy enemy, a thousand battles, a thousand victories.”

While we may not be at war, but the concept still applies. The better you know your capabilities, and your users, the better you’d be able to come up with an email campaign.

You can easily set up your test audience if you use email management software.

Email A/B test split

Unlayer Studio, for example, allows you to specify the percentage of subscribers that you want to do A/B tests on.

This step can become tedious if you don’t have an email marketing tool. Separating subscribers into A and B groups takes a lot of time manually. Plus, hand-picking subscribers minimizes the objectivity of A/B testing, which can hurt test accuracy.

What are some of the best practices you can follow?

Email marketing tools like Unlayer Studio, with drag-and-drop email builders, help speed up your email campaign process. They also contain useful tools to help you with A/B testing your email campaigns. However, the sword no matter how sharp is only as skilled as the one wielding it.

That’s why here we’ll walk you through various best practices you can use to make your email campaign more efficient.

Start with a game plan

For the highest chance of getting positive conversion results from an A/B test, you must have a hypothesis (game plan) to explain why one variation may perform better than the other.

Here are some example hypotheses you can use in your A/B testing:

If you’re testing multiple components, you’ll need to come up with a different null hypothesis for each of them.

Pick your battles

Whether you’re changing subject lines, button colors, or copy, you’ll probably have to run multiple A/B tests.

Here you need to prioritize which A/B tests are more important. Because the truth is not all A/B tests are created equal and nor should they be. To achieve the best results with the least effort, you want to prioritize the ideas that make the most sense.

ICE score can help you achieve this. According to business guru Sean Ellis, the ICE score can be used to grade your A/B testing ideas and prioritize which to run first. ICE score depends on three things:

Keep a record

As a business, you’ll run multiple email campaigns. In fact, before and after any event, chances are you’ll need to send an email campaign. With that in mind, it wouldn’t make much sense to A/B test everything every time.

That’s why keeping a record of your previous email campaigns helps you come up with new ones without spending too much time on A/B testing.

Plus, A/B tests do not always yield a positive outcome. A few of your variations will lower conversions, while many won’t make a noticeable difference.

Making sure you use the knowledge you gain from each A/B test to improve your campaigns is imperative for success.

Final Words

Start A/B testing your email campaigns now for better engagement and higher click-through rates. The process is quite simple, come up with a hypothesis of what you want to test, set up your A/B test, send it to your subscribers, and wait for the results.

Even if you don’t see a significant increase in opens or click-throughs, you’ll still learn something about your audience and you can use that knowledge to create more effective future campaigns.

Here are some pointers on optimizing your email campaigns using A/B testing:

Did you enjoy reading this guide? Here’s another article on content marketing in 2021 and how it can help your business.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store