Shorter monthly newsletter: slightly higher CTR, not reliable
Test run byMarketingExperimentsResearch publisher
Commentary byLiftArchiveOur analysis, not theirsCutting article summaries and a recap from MarketingExperiments' monthly newsletter gave slightly higher clickthrough, but the gain was significant in only one of three months.
We treat it as inconclusive. MarketingExperiments (MECLABS) ran the test on its own Best of the Month send after the same change won for a weekly MarketingSherpa newsletter.
- Clickthrough was slightly higher for the shorter version. Flat overall.
- The difference was statistically significant in only one of three monthly sends.
- The same change won for MarketingSherpa's weekly newsletter, so the result did not carry over.
The change
Summaries and the clinic recap removed; upcoming clinic kept.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
The monthly Best of the Month newsletter included article summaries, a recap of the last web clinic and a preview of the next one.
Variant
The shorter version dropped the article summaries and the clinic recap, keeping only the description of the upcoming clinic.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Monthly readers see the email less often and may value the extra context the summaries gave.
- The clinic recap may have been useful content for this audience, not clutter.
- Three sends gave few chances to build a clear signal.
Caveats: Only three monthly sends were tested, and exact numbers appear only in images. The result is inconclusive, not a win.
Who ran it
MarketingExperiments ran this test and published the results (MarketingExperiments). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on MarketingExperiments →
How to run this test on your site
- Hypothesis
- If we shorten our monthly newsletter by removing summaries and recaps, then clickthrough will rise because the email is quicker to scan.
- Primary metric
- Unique clickthrough rate per send
- Guardrails
- Unsubscribe rate
- Clicks on the upcoming event or webinar
- Who should test it
- Lead-gen and B2B teams with a monthly roundup email. Expect a smaller effect than on frequent sends, and plan enough sends to reach significance.
Questions
Can I reuse a winning email test on another list?
Retest it. This change won for a weekly list but was flat on a monthly one.
Why might a shorter email not win?
Readers of less frequent emails may want more context.
Cutting it can remove the content they came for.