Email A/B testing means sending two or more versions of one email to random slices of your list and comparing a single outcome, such as clicks or orders, to see which version works better.
The most useful email tests change the offer, the relevance or the subject line's promise, and they judge the winner on clicks or revenue rather than opens, which privacy features now inflate.
Below is how email split testing works, what real tests in our database found, and how to run subject line tests that you can trust.
Every result is linked to the test page and credited to whoever ran it.
How email split testing works
An email split test is a randomized experiment like any other A/B test.
Each subscriber gets one version, assignment is random, and you compare the versions on one metric you picked before sending.
Most email platforms offer two setups. In the first, you send the variants to a test slice of the list, wait, and send the winner to everyone else.
In the second, you send all variants to the whole list at once and learn for next time.
Mailchimp's help docs, for example, say you can test the subject line, From name, content or send time, with up to 3 variations per test.
It can pick a winner automatically by open rate, click rate or total revenue, or you can choose manually.
Send-time tests have to go to the full audience.
Klaviyo reports a win probability for campaign A/B tests and tags a result as statistically significant when that probability is above 90% and each variation has more than 50 deliveries.
Treat that tag as a minimum, not proof.
Fifty deliveries are far too few to detect the lifts most emails produce.
Why open rate is a weak winning metric
Apple's Mail Privacy Protection downloads remote content in the background by default, regardless of whether you engage with the email.
Open tracking relies on a remote pixel, so many Apple Mail users register as opens even when nobody read the message.
That adds noise to every open-rate test and can hide real differences.
Use click rate, conversion rate or revenue per recipient as the deciding metric when you can.
If you must test on opens, check that clicks moved in the same direction.
Our guide to A/B testing metrics covers how to pick a primary metric.
How big your list needs to be
Email tests use the same sample size math as website tests.
For two proportions, the recipients needed per variant at 95% confidence (two-sided) and 80% power is roughly:
n = (1.96 + 0.84)^2 × [p1(1 − p1) + p2(1 − p2)] / (p2 − p1)^2
Here p1 is your current rate and p2 is the rate you want to be able to detect.
The gap between them is your minimum detectable effect.
| Metric | Baseline (p1) | Detect (p2) | Relative lift | Recipients per variant |
|---|---|---|---|---|
| Click rate | 2.0% | 2.4% | +20% | about 21,100 |
| Click rate | 2.0% | 3.0% | +50% | about 3,800 |
| Open rate | 30% | 33% | +10% | about 3,800 |
Worked example for the first row: 7.84 × (0.0196 + 0.0234) / 0.004^2 = 7.84 × 0.0430 / 0.000016, which is about 21,100 people per variant.
A list of 10,000 split 10/10/80 gives each test cell 1,000 people, which can only detect very large click differences.
That is why "send to 20%, pick a winner after two hours" often crowns noise.
If your list is small, send the test to the whole list, test bigger changes, or repeat the same test across several sends.
Run your own numbers in the sample size calculator and check results with the significance calculator.
This math is the wrong tool for revenue per recipient, which is a skewed average rather than a rate.
A few large orders can swing it, so you need more recipients or a test built for means.
It also breaks if you compare three or more variants without adjusting for multiple comparisons.
A/B testing email marketing: what real tests found
Our database holds dozens of email tests, mostly from lead-gen and SaaS senders.
The pattern is consistent: changes to relevance and offer move results far more than cosmetic tweaks.
You can browse them all in the test database.
MarketingExperimentsMarketingExperimentsRelevance beat generic content
HomeAdvisor replaced three generic follow-up emails with three matched to the subscriber's remodeling project.
The project-matched series got 2x the conversion rate and 1.5x the clickthrough rate.
The Global Leadership Summit named the host site nearest each past attendee and got 87 more registrations than the general version.
The result was reported as a count, with no rate or significance.
Clarity of the offer and the ask
In a MarketingExperiments test on a rented list, an email with a clearer value proposition and a lower-commitment call to action got 104% more clicks.
For Sermo, MarketingExperiments found that offering several articles in one email lifted email capture by 197%, though only among people who clicked twice.
Small, specific fixes can count too. Adding one line with the start time in UTC to a webinar invite raised unique clickthrough by 2.7% and total clickthrough by 45%, both at 95% significance, in a MarketingExperiments test.
The metric you pick changes the winner
MECLABS Institute tested donation emails for Consumer Reports sent to about 450,000 subscribers.
The versions explaining member benefits lost on clickthrough but won 32% more revenue per donation and 16% more total revenue.
Judged on clicks alone, the better email would have been thrown out.
EnergySage signed its verification email off with a real team member's name and saw 3.98% more approved properties.
They measured a business outcome downstream of the email, not opens.
Length: no universal answer
Short versus long copy is the most common email test, and our results go both ways.
M7 Corporation's long-copy white paper email was about 3x more effective than short copy with enterprise developers, and the company warned it might not generalize.
MarketingExperiments' short vs long newsletter test for MarketingSherpa showed no significant difference in unique clicks.
A shorter monthly newsletter won in only 1 of 3 sends, even though the same change had worked for a weekly newsletter.
Test length with your own audience instead of copying a rule.
Some tests lose, and that is useful
For Consumer Reports, four single-value-point donation emails all underperformed the control.
Dashlane added customer reviews to a heavily optimized upgrade email and got about a 1% lift. Mature emails often have little headroom left.
CNET's welcome email went from a 32% to a 46% open rate and from 12% to 21% clickthrough, but that came from five treatments in a row.
It shows what iteration can do, not which single change worked.
Subject line A/B testing
Subject lines are the easiest email element to test, which is why most teams start there.
The tests in our database suggest the subject's promise matters more than its wording tricks.
What subject line tests in our database show
AppVault swapped a first-name subject line for one naming the applicant tracking system already on the prospect's site.
Unique opens went from 13% to 66%. The source gives no sample size, it measured opens rather than meetings, and it ran in 2016, before Apple's open-tracking change.
The Global Leadership Summit found that '13 world-class speakers' beat one star name repeatedly, though the case study gives no numbers.
For VolunteerMatch, MECLABS Institute found subject lines about network size lost 12.5% on click rate against passion-focused lines.
Apollo Education Group added the subscriber's name to a birthday email's preheader and got a 9% higher open rate.
An earlier run of the same test did poorly, which is a reminder that one email result can be noise.
How to run a subject line test you can trust
- Test a different promise, not a synonym. Relevance (AppVault's ATS name) and framing (breadth vs a star) produced the clear differences. Our headline guide covers the same principles.
- Judge on clicks, not just opens. A subject that wins opens but loses clicks is overpromising.
- Size the test before sending. Detecting a 10% relative open-rate lift from 30% needs about 3,800 people per variant.
- Change one thing. If the subject and preheader both change, you won't know which mattered.
- Repeat winners. Run a winning pattern on two or three more sends before treating it as a rule, as Apollo's retest shows.
- Log every result. Losses and ties, like the VolunteerMatch B2B email that got 75% more clicks but was not significant, stop you re-testing the same idea.
If your goal is growing the list rather than improving sends, see our guide to email capture, which draws on popup and form tests from the database.