A/B Testing in Email Marketing: What to Test and How to Measure Results

What to Test and How to Measure Results

Most email marketers work on assumptions. They think their readers prefer a short email over a long one. They believe a conversational tone is more appropriate than a professional one. They think the best time to send is Tuesday morning. They build campaigns based on intuition, articles from trade publications, and what worked for someone else in a completely different niche, and are surprised when their results plateau.

A/B testing removes the guesswork. It replaces assumptions with evidence, opinions with data, and stagnation with constant improvement. It’s the most dependable method to understand what your specific audience responds to, and it’s far more accessible than most beginners think.

Here’s the lowdown on what A/B testing is, why it’s important, what to test, how to test correctly, and how to interpret the results in a way that actually moves the needle on your email performance.

What Is A/B Testing in Email Marketing?

A/B testing is the process of sending two or more versions of an email to different segments of your list to see which version performs better. In the process, you change one variable between the two versions while keeping everything else the same. The results tell you which version your audience prefers.

For example:

• Version A: You send to 20% of your list.
• Version B: You send to another 20% of your list.
• After a set time period, the platform measures performance, usually by open rate or click rate.
• The winning version is automatically sent to the remaining 60%.

One A/B test isn’t sufficient. Several tests have to be conducted repeatedly over time. Every test teaches you something of value. And each lesson makes the next campaign more astute. Those small improvements compound over weeks and months, leading to better performance on all the important metrics.

Why A/B Testing Is Non-Negotiable for Serious Email Marketers

Let’s say you send out an email campaign and your open rate is 28 percent. Is that okay? Is that bad? Is it improved? You really don’t know — because you have nothing to compare it to other than industry averages, which are taken from entirely different businesses, different audiences, and different kinds of emails.

With A/B testing, you get a benchmark that really matters: your own performance, tested against your own audience.

Here is the power of testing beyond benchmarking:

Your audience is unique

No two email lists are the same. A subject line formula that kills it for a fitness newsletter might bomb completely for a B2B software brand. The only way to know what works for your readers is to test with them.

Small changes yield outsized results

A 5% increase in open rate seems small — but on a list of 10,000 subscribers, that’s 500 more people reading your email every single send. If you scale that over a year of campaigns, the compound effect is big.

It removes internal arguments

Teams, discussions about tone, design, and messaging can go on and on. A/B testing puts an end to those debates with data. It turns “our audience likes X” from an opinion to a fact.

It shows surprises

Some of the most counterintuitive findings are from A/B tests — the ugly plain-text email outperforms the beautifully designed HTML version; the longer subject line beats the short one; the email sent on Sunday evening gets the highest open rate. You never get these things without the testing.

The Golden Rule: Test One Variable at a Time

Before we move on to what to test, this rule warrants its own section because it is the most broken principle in A/B testing, and the one that makes or breaks the validity of your results.

If you change the subject line, send time, and call-to-action in one test, and Version B wins over Version A, you don’t know which change drove the improvement. Was it the subject? The time? The CTA?  All three of them? None of them on their own?

The test tells you something happened, but it can’t tell you what happened — which means you can’t reliably apply the lesson to future campaigns.

One test.  One variable.  One lesson, one step at a time. It may seem slower, but the knowledge you gain is clear, reliable, and truly actionable.

What to Test: The Complete A/B Testing Menu

You can test dozens of variables in an email campaign. Here’s a detailed breakdown, organized by category, with tips on what to look for while assessing findings.

1. Subject Lines

Subject lines are the most frequently tested aspect of email marketing, and for good reason. They are the single most important driver of open rates, and even little increases in open rates have a cascading influence on all downstream metrics.

What to test within subject lines:

• Length: Short (fewer than 40 characters) vs. lengthy (more than 60 characters). Conventional wisdom supports short subject lines, but many audiences prefer descriptive, particular subject lines that provide a clear reason to open.
• Tone: Curiosity-driven (“I wasn’t supposed to share this”) versus direct advantage (“Cut your editing time in half with this tool”). These two approaches appeal to distinct psychological impulses and perform differently among audiences.
• Personalization: Subject line with recipient’s first name versus without. Personalization typically increases open rates, but not always. Some people find it gimmicky. Try it with yours.
• Question vs. statement: “Are you making these email mistakes?” versus “5 email mistakes costing you subscribers.” Questions stimulate the reader’s brain differently from declarative sentences.
• Urgency against no urgency: “Sale ends tonight” against “Our best deal of the year.” If your offer truly has a deadline, leading with urgency almost always boosts performance, but only if it’s genuine.
• With or without emoji: A suitable emoji can improve the visual contrast in the inbox. Whether it helps or harms is strongly influenced by your brand voice and target audience demographics.

Primary metric to track: Open rate.

2. Preview Text

The preview text, which displays next to or below the subject line in most inboxes, is one of the most underutilized testing possibilities in email marketing. Most marketers simply disregard it or leave it as the default placeholder text.

What to test in preview text:

• Preview text that extends the subject line’s hook vs. preview text that introduces a separate, complementary idea
• A direct benefit statement vs. a curiosity gap
• A question vs. a bold claim
• Including a CTA hint in the preview text vs. keeping it mysterious

Remember, the subject line and preview text work together. Test them together, not individually.

Primary metric to track: Open rate.

3. Send Time and Day

When you send an email, it can be as important as what’s inside it. The appropriate send time ensures that your email arrives at the top of your subscriber’s inbox when they are most likely to check it and least likely to be distracted.

What to test in send time and day:

• Day of week: Tuesdays, Wednesdays, and Thursdays are the most suggested days, but also the most competitive. Testing on Monday or Sunday may indicate an underserved time for your target.
• Time of day: Morning (7-9 AM), midday (12-1 PM), and evening (6-8 PM)—these windows represent various subscriber behaviors and contexts.
• Time zone targeting: Sending at 8 AM in the subscriber’s local time zone vs. a fixed time for your entire list.

One major caveat: send time optimization typically yields smaller increases than subject line or content testing. It is worth testing, but it is usually not the most important variable on your list.
Primary metric to track: Open rate and click-through rate.

4. Email Copy and Length

When a subscriber opens an email, the copy takes over. Even if your open rates remain constant, testing your body copy can significantly increase click-through rates and conversions.

What to test in email copy and length:

• Length: Short and snappy (150-300 words, one main point, one CTA) vs. long-form (600-900 words, additional context, storytelling, and depth). Neither is universally superior. Longer emails work effectively with some audiences, especially those who subscribe to newsletters and educational content. Others, especially in e-commerce or transactional environments, value brevity and a quick path to action.
• Tone: Formal and professional versus conversational and relaxed. This is generally associated with audience demographics and the character of your brand, although assumptions can be incorrect. Test it.
• Storytelling vs. direct: An email that begins with a personal narrative and then moves on to the offer, as opposed to one that starts with the benefit or news.
• Plain text against HTML: A simple, unformatted text-only email might feel more personable and authentic, and it frequently generates more engagement than a finely crafted HTML template. This is one of the most unexpected findings in email testing, yet it remains true across sectors.
• First line: The first sentence of your email is critical. It influences whether the reader continues or closes the tab. Compare a question opener to a bold statement vs. a story opening.
• Primary metric to track: Click-through rate (CTR), scroll depth (if trackable), and conversion rate.

5. Call-to-Action (CTA)

Your CTA is where intrigue turns into action. Even minor adjustments to your call-to-action might cause large swings in conversion rates.

What to test in CTA:

• Button versus text link: A large styled button versus a hyperlinked sentence in the body copy. Buttons work effectively in commercial emails, although plain text links feel more natural in conversational or newsletter formats.
• CTA copy: “Shop now,” “See the collection,” and “Claim your discount” portray varying degrees of commitment and urgency. The first-person phrase (“Get my free guide”) often surpasses the second-person (“Get your free guide”).
• CTA positioning: CTAs can be placed above the fold (visible without scrolling), at the conclusion of the email, or throughout the body.
• Number of CTAs: One concentrated CTA versus two or three alternatives. Single CTAs tend to convert better than multiple CTAs in offers and sales emails. Multiple CTAs might be useful in digest-style mailings where you link to multiple pieces of content.
• CTA color and size (HTML emails): Visual prominence has a substantial impact on click behavior.

Primary metric to track: Click-through rate and conversion rate.

6. From Name and Sender Identity

Who the email appears to be from has an unexpected impact on open rates—perhaps as much as the subject line for established lists.

What to test in [From Name and Sender Identity]:

• Personal names vs. brand name: “James” or “James from Copycraft” vs. “Copycraft Newsletter”. Personal sender names frequently outperform brand names because they feel more personable and less like a mass mailing.
• Different personal names: If your brand has many team members or voices, test which name resonates most with your target audience.
• Email address: hello@yourbrandname.com vs. newsletter@yourbrandname.com vs. firstname@yourbrandname.com. No-reply addresses (noreply@yourbrand.com) typically underperform, so avoid them totally.

Primary metric to track: Open rate.

7. Offers and Incentives

If you send promotional emails, you should evaluate the structure of your offer as well as how you present it.

What to test in offers and incentives:

• Discount types: Percentage discount (“20% off your order”) versus a fixed value amount (“Save $15 today”). Research consistently shows that dollar amounts feel more tangible and outperform percentages for lower-priced items, whereas percentages work better for high-ticket items.
• Free shipping vs. discount: Many e-commerce studies demonstrate that free shipping is a stronger motivator than an equivalent percentage discount.
• Urgency framing: “Sale ends Sunday” versus “Only 48 hours left” versus “Last chance.”
• Exclusivity framing. “For subscribers only” vs. “Weekend sale — shop now.”

Primary metric to track: Conversion rate and revenue per email.

8. Images and Visual Elements

Visual features in HTML emails have a major impact on engagement, but not always in the way you would think.

What to test in images and visual elements:

• Image-heavy vs. sparse images: More visuals versus a cleaner, text-focused layout.
• Hero image vs. no hero image: A large banner image at the top of the email vs. jumping straight into the copy.
• Product images: Lifestyle photography (product in context) vs. clean product pictures with white backdrops.
• GIFs: When used correctly, animated GIFs can enhance engagement and click rates, but they also increase email file size, which can affect load speed and spam filtering.

Primary metrics to track: Click-through rate and, for e-commerce, conversion rate.

How to Run a Proper A/B Test: A Step-by-Step Guide

Knowing what to test is only half the battle. Run your tests right and your results will be reliable.

Step 1: Make a hypothesis

Each test should start with a hypothesis: not just, “Let’s try a different subject line,” but, “I believe a curiosity-driven subject line will beat a direct-benefit subject line because our audience tends to engage more with educational content.” This is a discipline where you have to think clearly about why you are testing something, and that makes the results more useful.

Step 2: Isolate the variable

Change one thing and only one thing from Version A to Version B. Everything else—send time, list segment, email copy, CTA—stays the same.

Step 3: Determine your sample size

You normally require a minimum of 1,000 subscribers per variant for statistically meaningful results, so at least 2,000 subscribers for a standard A/B test. If your list is smaller, your results may be directionally interesting, but not statistically reliable. If you have fewer than 1,000 total on your list, run the tests anyway and look for patterns over time instead of trying to draw conclusions from any one test.

Step 4: Set a clear success metric before you send

Decide in advance what winning is. Are you optimizing for open rate? Click rate? Revenue? And don’t change the success metric after you see the results; that’s a form of data cherry-picking that leads to false conclusions.

Step 5: Let the test run long enough

Most email platforms provide a testing window before declaring a winner (usually 2, 4, or 24 hours). A 4-hour window will be sufficient to gather meaningful data for most campaigns. A longer window may be needed for sends to global audiences in multiple time zones.

Step 6: Wait for statistical significance

Most platforms show a confidence level along with the test results. Aim for 95% or higher statistical significance before you call a winner. If it is only 70% confident, it could just be random variation, and it would be misleading to act on it.

How to Measure and Interpret Your Results

Running the test is one thing. Knowing what the results mean and what to do next is where most marketers fall short.

Core metrics and what they tell you:

• Open rate is a measurement of how successful your subject line is, how well people recognize the sender’s name, and how well you choose your send time. It is the right measure for testing any of these variables.
• Click-through rate (CTR) shows how engaging your email content and CTA are after someone opens it. If your open rate is high, but your click-through rate is low, you’ve written a good subject line, but your email isn’t delivering on the promise.
• Click-to-open rate (CTOR) is the percentage of openers who clicked, not the percentage of all recipients. This is arguably more useful than raw CTR as it separates the performance of your content from the performance of your subject line.
• Conversion rate tells you how many of those who received the email took the desired action – a purchase, a sign-up, a download. This is the ultimate metric for promotional campaigns and should always be tracked, even if it is not your primary test metric.
• Revenue per email (RPE) is the total revenue generated from an email campaign divided by the total number of emails sent. This is the most straightforward metric for email performance for e-commerce and product-based businesses.
• Unsubscribe rate is often an overlooked metric, but very informative. A spike in unsubscribes after a particular type of email or subject line tactic is strong feedback that you’ve misread your audience’s expectations.

Building a Testing Culture: The Long Game

The best email marketers don’t run the occasional A/B test; they create a systematic culture of continuous testing. Here is what it looks like in practice:

Maintain a testing log

Write down every test you run. The hypothesis. The variable that was tested. The results. The conclusion. Over time, this log becomes an invaluable reference—a map of your audience’s preferences based on real evidence.

Create a swipe file of winners

If a subject line formula, CTA wording, or tone keeps performing better, keep it archived. These are your proven assets and should form the basis of future campaigns.

Return to old tests

Audiences evolve. As your list grows and changes, the list you ran eighteen months ago may have different results today. Don’t presume that old results are eternally correct. Re-testing them periodically keeps your knowledge current.

Sort tests by potential impact

Not all testing is the same. Subject line tests usually produce the most important results, as they impact all downstream metrics. Start with that, and then work your way down the list to lower-leverage variables such as image placement or CTA button color.

Expect to be surprised by some of the tests

The most valuable tests are usually the ones that disprove your assumptions. If your carefully crafted “better” version gets beaten by a “worse” version, don’t dismiss it. Be curious instead. What did you learn about your audience that you didn’t already know?

Common Mistakes to Avoid in A/B Testing

These mistakes can make your results invalid or lead you to the wrong conclusions, even with the best intentions.

• Testing too many variables at one time. Already covered, but worth repeating because it is the most common mistake.
• The test finished before time. What looks like a winner after one hour can be losing by hour four. Give your tests a little breathing space before you declare a winner.
• Ignoring the sample size.  But acting on results from a test with only 200 subscribers per variant is like reading the weather from one thermometer in one room. The sample is too small to be reliable.
• Letting the test overlap with big events. If you run your A/B test during a holiday, product launch, or some other strange time, your results will be skewed. The external event is a confounding variable.
• Not responding to results. The whole point of testing is to make use of what you learn. Jot down what you find. Change your default approach based on winners. Let the lessons you learn guide your future campaigns.

Final Thoughts: Test Your Way to Excellence

A/B testing isn’t just for enterprise marketing teams with dedicated data analysts. Anyone with an email list, a curious mind, and the discipline to change one thing at a time can do this.

The marketers who occasionally outperform the industry averages aren’t necessarily more talented or more creative than their peers. They are more systematic. They treat their email list like a laboratory. They ask better questions, collect real evidence, and let their audience guide them to better results.

Each test you run is an investment. The returns are compounding and irreversible. Start with your subject line, run through the process, track your results, and leverage that info. You won’t have to guess what works for your audience over time. You’ll know.

The best email marketing strategy is not copied from someone else’s case study; it is built from your own data. Run your first A/B test on your next campaign and let your audience write the playbook.