Skip to main content

Getting started with email split testing can be daunting if you’re not quite sure what to do. You know you need to have a significant sample size, but how big should it be?  Should the two versions have a small difference, or a radical difference?

To set your mind at ease we’ve put together a list of things you need to consider for email split tests.

1. Do start with a hypothesis

Know what you’re testing and why. Have a clear objective from the start and you’ll better understand the impact of your test.

Dobbies Garden Centre’s hypothesis was that adopting a mobile responsive template would get better results. They tested it against their existing template and saw a 400% increase in click-to-open rates.

2. Do have the right sample size

Your sample needs to be significant in terms of numbers or percentage size. This is the most important thing to get right – otherwise your data won’t be relevant, and you won’t know if your results are just down to a random variation. This calculator can help you determine what you need.

3. Do watch your timing

Send both versions simultaneously, unless you’re testing different send times. Otherwise, changing the time would add a second variable to your split testing and skew the results.

4. Do understand what you’re testing

Know which element your test actually affects. Changing an image? Measure click-throughs, not open rates. On the other hand, testing the subject line, sender name and pre-header text will most likely impact your open rates.

5. Do base the test on one change

It’s very important not to change more than one element at a time between your versions. Having three or more variants of colors in a call-to-action button is a correct split test, but changing the copy as well as the colors is not.

The reason for this is changing more than one element at a time will make result attribution impossible. That’s why it’s also more reliable to test one word change at a time in a subject line or headline, rather than change the whole sentence. It’s trickier with a tone of voice test, but we recommend you keep the message similar to have some consistency.

If you want to test multiple elements, you need to use the multivariate testing methodology explained later in this guide.

6. Don’t muddle your variables

Be clear about what you’re testing:

7. Don’t stop testing

Even if you think you’ve found the perfect combination, you still need to carry on testing to see how things change.

After all, your business doesn’t stay the same – and neither will your customers! You’ll have new products, promotions and messages. Their interests and finances may change, and the novelty of your latest enhancements may wear off after a while.

Download our ‘Email Testing Planner’, and get more advice on what makes great tests from our new guide ‘Optimize your Email Campaign Performance with Testing’.