B2B email led by proof vs network size: +75% clicks, not significant
Commentary byLiftArchiveOur analysis, not theirsA B2B email that led with proof that employee volunteering pays off got 75% more clickthroughs than one leading with network size, but we count it as inconclusive because the result was not statistically significant.
MECLABS Institute ran the test with VolunteerMatch during a live session at MarketingSherpa Email Summit 2015, and it was published as a MarketingSherpa case study.
The list was small and open rates did not differ.
- +75% clickthrough for the proof version versus the network version. Treated as flat: the difference was not statistically significant.
- Open rates showed no significant difference between subject lines.
- A third, ease-of-use version could not be compared, so its relative difference was not computed.
- The rest of each email, including the two CTAs, was the same.
The change
Only the headline and opening change, from network size to proof.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
An email to VolunteerMatch's B2B list whose subject line, headline and opening paragraph led with the size of its nonprofit network.
Variant
The same email, but the subject line, headline and opening paragraph led with proof that employee volunteering programs make a difference. The Summit audience helped write it.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Program managers often struggle to prove value to their bosses, so proof may speak to a real pain.
- The B2B list was small, so even a large-looking gap can come from chance.
- An earlier test on the volunteer list also found network size was the weakest theme, which points the same way but doesn't confirm it.
Caveats: Small list and not statistically significant, so we treat it as inconclusive even though the source named proof the winner. The data is from early 2015.
Who ran it
MECLABS Institute ran this test and published the results (MarketingSherpa case study). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on MECLABS Institute →
How to run this test on your site
- Hypothesis
- If we open our B2B emails with proof that our product delivers results instead of how big it is, then more buyers will click because they need evidence to win internal buy-in.
- Primary metric
- Clickthrough rate
- Guardrails
- Open rate
- Unsubscribe rate
- Demo or contact requests from clicks
- Who should test it
- B2B SaaS companies whose buyers must justify spend to managers. With a very small list, expect to repeat the test or pool several sends before you can trust the result.
Questions
What should a B2B email lead with?
Often the buyer's biggest worry. For VolunteerMatch, proof that the program pays off looked stronger than network size, though the result wasn't significant.
How big does an email list need to be for A/B testing?
Big enough that each group gets a meaningful number of clicks.
With small lists, large gaps like 75% can still be chance.
Why can a 75% lift be inconclusive?
If only a few people click in each group, a small difference in counts looks like a big percentage.
Significance testing checks whether the gap is likely real.