How to utilize A/B testing to benefit your email marketing campaigns

Performing an A/B test is common to email marketers to ensure that the campaigns sent to customers are the best performers.

However, the way you conduct your A/B test in your marketing automation software is crucial. Wrong execution could bring more harm than good, and this is the last thing you would want to happen.

The most common question of email marketers is “How long should you run your A/B test?” This is understandable because time is important and waiting could be a problem.

The truth is, time is just one of the many factors you need to consider. That is what we want you to realize in this post as we have an in-depth discussion on how to run your A/B test the right way and how long should it be running.

💔 Falling out of love with your clients? Trade some of your worst clients for the best companies in the world with SolidGigs, our premium weekly freelance job list & course library. Love your business again. Learn more »

What is A/B testing?

To give you a quick review, A/B testing in email marketing means producing two versions of your email campaign, by changing one variable to know which would bring you better results.

It could be your subject line, image, or CTA (call to action). Both versions must be sent to an ideal sample size that is representative of your audience. The winning version will be blasted to the entire list.  

There are three things you need to identify before running your A/B test: goal, sample size, and time. Let me explain to you how these things are connected to each other.  

Start by determining your goal

The length of time that you must allocate for A/B testing must be guided by your goal. Ask yourself: Which variable would you like to check? Not all variables need the same amount of time for testing.

For instance, an A/B test that examines which subject line can bring higher open rates can have a shorter running time compared to an A/B test aimed at knowing which email can bring higher revenues.

Enter your email. Grow your business.

Submit your email below and join 45,000+ creatives who get our most helpful content via email every week. 100% free. Unsubscribe anytime. Privacy protected.

Why? It’s because the time people need to think about your offer is longer compared to the time they need to assess whether your email is worth clicking or not.  

Later, we will tell you the ideal time to allocate based on your goals. For now, you must know that having enough sample size is also important to guarantee the validity of your results.

Generating the ideal sample size

The tricky part of doing A/B testing with emails is that you have a finite audience. Unlike social media ads or landing pages where you can gather new audience continuously, your email list has a definite size.

This means that once you’ve sent your emails, there is no way you can add new contacts to the test.

We all know that a small sample size is never recommendable, as it is not representative of your audience, which could lead to invalid and unreliable results. Remember, the larger the sample size, the more accurate the results are.

Also, you would rather send your campaign to your small email list than bother running tests.  That would save you more time.

Ideally, your email list must have at least 1,000 contacts to generate a sample size. You can use sample size calculators such as Optimizely to determine the ideal sample size for your test.

Often, the larger the sample size, the longer you should run your A/B test to give more time to your wide audience.

How long should you be running your A/B test?

Different email marketers would run their test in various ways. Here, I will give you a variety of ways on how to determine the length of time for your A/B test based on our observations.

Earlier I mentioned about setting your goals and determining what you want to achieve. Are you looking at opens or revenues?

Ideally, you would run a test for open rates for 2 hours, as customers do not need much time to determine if your email is worth their time or not based on your subject line.

For revenues, I recommend that you allocate 12 hours since more time is needed for customers to arrive at a decision.

You also have the option to review your analytics so that you can use your own data as a benchmark. For instance, if your contacts take three hours on average to open your past emails, then you can allocate 3 hours when running an A/B test for subject lines.

If time is not an issue, then it’s advisable to run your A/B test for the full business cycle, which is often one week. Remember, a lot of people comb through their inbox at least once a week. It only makes sense to give them enough time to open your email.

Proper execution of A/B test is important to make sure that you are not following the wrong data. Again, take note of these three things: goal, sample size, and time. So, what are you waiting for? Create your email campaign and start testing!

Comment with your thoughts about A/B testing!

Keep the conversation going...

Over 5,000 of us are having daily conversations over in our free Facebook group and we'd love to see you there. Join us!