A/B Testing in Email Marketing: What to Test and How to Measure Results

What to Test and How to Measure Results

Most email marketers work on assumptions. They think their readers prefer a short email over a long one. They believe a conversational tone is more appropriate than a professional one. They think the best time to send is Tuesday morning. They build campaigns based on intuition, articles from trade publications and what worked for someone else in a completely different niche, then wonder why their results plateau.

A/B testing removes the guesswork. It replaces assumptions with evidence and opinions with data. It’s a practical way to understand what your specific audience responds to, and it’s far more accessible than most beginners think.

Here’s the lowdown on what A/B testing is, why it’s important, what to test, how to test correctly and how to interpret the results in a way that can improve your email performance.

What Is A/B Testing In Email Marketing?

A/B testing is the process of sending two or more versions of an email to different segments of your list to see which version performs better. In the process, you change one variable between the two versions while keeping everything else the same. The results tell you which version your audience prefers.

For example:

• Version A: You send to 20% of your list.
• Version B: You send to another 20% of your list.
• After a set time period, the platform measures performance, usually by open rate or click rate.
• The winning version is automatically sent to the remaining 60%.

One A/B test isn’t sufficient. Several tests have to be conducted repeatedly over time. Every test teaches you something. Each lesson can inform the next campaign, and those small improvements can add up over weeks and months.

Why A/B Testing Is Non-Negotiable For Serious Email Marketers

Let’s say you send out an email campaign and your open rate is 28 percent. Is that okay? Is that bad? Has it improved? You really don’t know because you have nothing to compare it to other than industry averages, which come from different businesses, audiences and types of emails.

With A/B testing, you get a benchmark that really matters: your own performance, tested against your own audience.

Here is the value of testing beyond benchmarking:

Your audience is unique

No two email lists are the same. A subject line formula that works well for a fitness newsletter might bomb completely for a B2B software brand. The only way to know what works for your readers is to test with them.

Small changes can yield noticeable results

A 5% increase in open rate seems small, but on a list of 10,000 subscribers, that’s 500 more people reading your email every single send. If you scale that over a year of campaigns, the effect can add up.

It removes internal arguments

Teams can spend a long time discussing tone, design and messaging. A/B testing gives those discussions some evidence to work with. It turns “our audience likes X” from an opinion into something you can test.

It shows surprises

Some of the most counterintuitive findings come from A/B tests. An ugly plain-text email can outperform a beautifully designed HTML version. A longer subject line can beat a short one. An email sent on Sunday evening can get the highest open rate. You won’t discover these differences without testing.

The Golden Rule: Test One Variable At A Time

Before we move on to what to test, this rule warrants its own section because it is one of the most commonly broken principles in A/B testing, and breaking it can make your results difficult to interpret.

If you change the subject line, send time and call-to-action in one test, and Version B wins over Version A, you don’t know which change drove the improvement. Was it the subject? The time? The CTA? All three of them? None of them on their own?

The test tells you something happened, but it can’t tell you what happened. That means you can’t reliably apply the lesson to future campaigns.

One test. One variable. One lesson, one step at a time. It may seem slower, but the knowledge you gain is clearer and easier to apply.

What To Test: The Complete A/B Testing Menu

You can test dozens of variables in an email campaign. Here’s a detailed breakdown, organised by category, with tips on what to look for while assessing the findings.

1. Subject Lines

Subject lines are one of the most frequently tested aspects of email marketing, and for good reason. They have a direct influence on open rates, and even small increases in open rates can affect downstream metrics.

What to test within subject lines:

• Length: Short (fewer than 40 characters) vs. lengthy (more than 60 characters). Conventional wisdom supports short subject lines, but many audiences prefer descriptive, specific subject lines that provide a clear reason to open.
• Tone: Curiosity-driven (“I wasn’t supposed to share this”) versus direct advantage (“Cut your editing time in half with this tool”). These two approaches appeal to distinct psychological impulses and can perform differently among audiences.
• Personalisation: Subject line with the recipient’s first name versus without. Personalisation can increase open rates, but not always. Some people find it gimmicky. Try it with your own audience.
• Question vs. statement: “Are you making these email mistakes?” versus “5 email mistakes costing you subscribers.” Questions engage the reader differently from declarative sentences.
• Urgency against no urgency: “Sale ends tonight” against “Our best deal of the year.” If your offer truly has a deadline, leading with urgency can improve performance, but only if the deadline is genuine.
• With or without emoji: A suitable emoji can improve the visual contrast in the inbox. Whether it helps or harms depends heavily on your brand voice and target audience.

Primary metric to track: Open rate.

2. Preview Text

The preview text, which displays next to or below the subject line in most inboxes, is one of the most underused testing possibilities in email marketing. Many marketers simply disregard it or leave it as the default placeholder text.

What to test in preview text:

• Preview text that extends the subject line’s hook vs. preview text that introduces a separate, complementary idea
• A direct benefit statement vs. a curiosity gap
• A question vs. a bold claim
• Including a CTA hint in the preview text vs. keeping it mysterious

Remember, the subject line and preview text work together. Test them together when you want to assess the combination.

Primary metric to track: Open rate.

3. Send Time and Day

When you send an email can be as important as what’s inside it. The appropriate send time ensures that your email arrives near the top of your subscriber’s inbox when they are most likely to check it and least likely to be distracted.

What to test in send time and day:

• Day of week: Tuesdays, Wednesdays and Thursdays are commonly suggested days, but they are also competitive. Testing Monday or Sunday may reveal an underserved time for your target audience.
• Time of day: Morning (7-9 AM), midday (12-1 PM) and evening (6-8 PM) represent different subscriber behaviours and contexts.
• Time zone targeting: Sending at 8 AM in the subscriber’s local time zone vs. a fixed time for your entire list.

One major caveat: Send-time optimisation typically produces smaller gains than subject line or content testing. It is worth testing, but it is usually not the most important variable on your list.

Primary metric to track: Open rate and click-through rate.

4. Email Copy and Length

When a subscriber opens an email, the copy takes over. Even if your open rates remain constant, testing your body copy can increase click-through rates and conversions.

What to test in email copy and length:

• Length: Short and snappy (150-300 words, one main point, one CTA) vs. long-form (600-900 words, additional context, storytelling and depth). Neither is universally superior. Longer emails can work well with some audiences, especially those who subscribe to newsletters and educational content. Others, especially in e-commerce or transactional environments, value brevity and a quick path to action.
• Tone: Formal and professional versus conversational and relaxed. This is generally associated with audience demographics and the character of your brand, although assumptions can be incorrect. Test it.
• Storytelling vs. direct: An email that begins with a personal narrative and then moves on to the offer, as opposed to one that starts with the benefit or news.
• Plain text against HTML: A simple, unformatted text-only email might feel more personable and authentic, and it can generate more engagement than a finely crafted HTML template.
• First line: The first sentence of your email matters. It can influence whether the reader continues or closes the tab. Compare a question opener with a bold statement or a story opening.
• Primary metric to track: Click-through rate (CTR), scroll depth (if trackable) and conversion rate.

5. Call-To-Action (CTA)

Your CTA is where intrigue turns into action. Even minor adjustments to your call-to-action can cause changes in conversion rates.

What to test in CTA:

• Button versus text link: A large styled button versus a hyperlinked sentence in the body copy. Buttons can work well in commercial emails, although plain text links may feel more natural in conversational or newsletter formats.
• CTA copy: “Shop now,” “See the collection” and “Claim your discount” convey varying degrees of commitment and urgency. The first-person phrase (“Get my free guide”) can outperform the second-person (“Get your free guide”).
• CTA positioning: CTAs can be placed above the fold, at the conclusion of the email or throughout the body.
• Number of CTAs: One concentrated CTA versus two or three alternatives. Single CTAs can work better in offers and sales emails. Multiple CTAs might be useful in digest-style mailings where you link to multiple pieces of content.
• CTA colour and size (HTML emails): Visual prominence can affect click behaviour.

Primary metric to track: Click-through rate and conversion rate.

6. From Name and Sender Identity

Who the email appears to be from can have an unexpected impact on open rates, particularly for established lists.

What to test in From Name and Sender Identity:

• Personal names vs. brand name: “James” or “James from Copycraft” vs. “Copycraft Newsletter”. Personal sender names can outperform brand names because they feel more personal and less like a mass mailing.
• Different personal names: If your brand has many team members or voices, test which name resonates most with your target audience.
• Email address: hello@yourbrandname.com vs. newsletter@yourbrandname.com vs. firstname@yourbrandname.com. No-reply addresses (noreply@yourbrand.com) typically underperform, so avoid them where possible.

Primary metric to track: Open rate.

7. Offers And Incentives

If you send promotional emails, you should evaluate the structure of your offer as well as how you present it.

What to test in offers and incentives:

• Discount types: Percentage discount (“20% off your order”) versus a fixed value amount (“Save $15 today”). Research can vary by product and audience, so test both options rather than assuming one will always perform better.
• Free shipping vs. discount: Test free shipping against an equivalent percentage discount to see which matters more to your customers.
• Urgency framing: “Sale ends Sunday” versus “Only 48 hours left” versus “Last chance.”
• Exclusivity framing: “For subscribers only” vs. “Weekend sale. Shop now.”

Primary metric to track: Conversion rate and revenue per email.

8. Images and Visual Elements

Visual features in HTML emails can affect engagement, but not always in the way you would expect.

What to test in images and visual elements:

• Image-heavy vs. sparse images: More visuals versus a cleaner, text-focused layout.
• Hero image vs. no hero image: A large banner image at the top of the email vs. jumping straight into the copy.
• Product images: Lifestyle photography (product in context) vs. clean product pictures with white backgrounds.
• GIFs: When used correctly, animated GIFs can affect engagement and click rates, but they also increase email file size, which can affect load speed and deliverability.

Primary metrics to track: Click-through rate and, for e-commerce, conversion rate.

How To Run A Proper A/B Test: A Step-By-Step Guide

Knowing what to test is only half the battle. Run your tests correctly, and your results will be easier to interpret.

Step 1: Make A Hypothesis

Each test should start with a hypothesis: not just, “Let’s try a different subject line,” but, “I believe a curiosity-driven subject line will beat a direct-benefit subject line because our audience tends to engage more with educational content.” Thinking clearly about why you are testing something makes the results more useful.

Step 2: Isolate The Variable

Change one thing and only one thing from Version A to Version B. Everything else, including send time, list segment, email copy and CTA, stays the same.

Step 3: Determine Your Sample Size

You normally require a minimum of 1,000 subscribers per variant for statistically meaningful results, so at least 2,000 subscribers for a standard A/B test. If your list is smaller, your results may be directionally interesting but not statistically reliable. If you have fewer than 1,000 total subscribers on your list, run the tests anyway and look for patterns over time instead of trying to draw conclusions from any one test.

Step 4: Set A Clear Success Metric Before You Send

Decide in advance what winning means. Are you optimising for open rate, click rate or revenue? Don’t change the success metric after you see the results. That is a form of data cherry-picking that can lead to false conclusions.

Step 5: Let The Test Run Long Enough

Most email platforms provide a testing window before declaring a winner, usually 2, 4 or 24 hours. A 4-hour window may be sufficient for some campaigns. A longer window may be needed for sends to global audiences in multiple time zones.

Step 6: Wait for Statistical Significance

Most platforms show a confidence level along with the test results. Aim for 95% or higher statistical significance before you call a winner. If it is only 70% confident, it could be random variation, so it would be misleading to act on it as a firm result.

How to Measure and Interpret Your Results

Running the test is one thing. Knowing what the results mean and what to do next is where many marketers fall short.

Core metrics and what they tell you:

• Open rate is a measurement of how successful your subject line is, how well people recognise the sender’s name and how well you choose your send time. It is a useful measure for testing these variables.
• Click-through rate (CTR) shows how engaging your email content and CTA are after someone opens it. If your open rate is high but your click-through rate is low, you’ve written a good subject line, but your email isn’t delivering on the promise.
• Click-to-open rate (CTOR) is the percentage of openers who clicked, not the percentage of all recipients. This can be more useful than raw CTR because it separates the performance of your content from the performance of your subject line.
• Conversion rate tells you how many of those who received the email took the desired action, such as a purchase, sign-up or download. This is an important metric for promotional campaigns and should be tracked even if it is not your primary test metric.
• Revenue per email (RPE) is the total revenue generated from an email campaign divided by the total number of emails sent. This is a straightforward metric for measuring email performance for e-commerce and product-based businesses.
• Unsubscribe rate is often an overlooked metric, but it can be informative. A spike in unsubscribes after a particular type of email or subject line tactic can indicate that you’ve misread your audience’s expectations.

Building A Testing Culture: The Long Game

The best email marketers don’t run the occasional A/B test. They create a regular process for testing and learning. Here is what it looks like in practice:

Maintain A Testing Log

Write down every test you run. The hypothesis. The variable that was tested. The results. The conclusion. Over time, this log becomes a useful reference for understanding your audience’s preferences based on your own results.

Create A Swipe File Of Winners

If a subject line formula, CTA wording or tone keeps performing better, keep it archived. These proven examples can inform future campaigns.

Return To Old Tests

Audiences evolve. As your list grows and changes, the audience you tested eighteen months ago may produce different results today. Don’t presume that old results are eternally correct. Re-testing them periodically keeps your findings current.

Sort Tests By Potential Impact

Not all testing has the same potential impact. Subject line tests can produce important results because they affect whether people open the email in the first place. Start there, then work your way down to variables such as image placement or CTA button colour.

Expect To Be Surprised By Some Of The Tests

The most useful tests are often the ones that disprove your assumptions. If your carefully crafted “better” version gets beaten by a “worse” version, don’t dismiss it. Ask what it tells you about your audience that you didn’t already know.

Common Mistakes To Avoid In A/B Testing

These mistakes can make your results invalid or lead you to the wrong conclusions, even with the best intentions.

• Testing too many variables at one time. Already covered, but worth repeating because it is one of the most common mistakes.
• The test finished too early. What looks like a winner after one hour can be losing by hour four. Give your tests some breathing space before you declare a winner.
• Ignoring the sample size. Acting on results from a test with only 200 subscribers per variant is like reading the weather from one thermometer in one room. The sample is too small to be reliable.
• Letting the test overlap with big events. If you run your A/B test during a holiday, product launch or some other unusual period, your results may be skewed. The external event is a confounding variable.
• Not responding to results. The whole point of testing is to use what you learn. Jot down what you find. Change your default approach based on winners. Let the lessons you learn guide your future campaigns.

Final Thoughts: Test Your Way to Excellence

A/B testing isn’t just for enterprise marketing teams with dedicated data analysts. Anyone with an email list, a curious mind and the discipline to change one thing at a time can do this.

The marketers who occasionally outperform industry averages aren’t necessarily more talented or more creative than their peers. They are more systematic. They treat their email list like a laboratory. They ask better questions, collect real evidence and let their audience guide them towards better results.

Each test you run adds to your understanding of your audience. Over time, those lessons can compound. Start with your subject line, run the process, track your results and use what you learn. You won’t have to keep guessing what works for your audience. You’ll have evidence to guide your decisions.

The best email marketing strategy is not copied from someone else’s case study. It is built from your own data. Run your first A/B test on your next campaign and let your audience help write the playbook.