Value proposition rewrite on product page: +445% lift not trusted
Test run byMarketingExperimentsResearch publisher
Commentary byLiftArchiveOur analysis, not theirsA rewritten value proposition on a top product page first showed a 445.6% conversion lift, but we treat it as inconclusive because a rerun showed the numbers were noise.
MarketingExperiments (MECLABS) ran the test for a seller of mostly digital products to high-tech industries.
In the follow-up, two identical copies of the control differed by 30%.
- First run: Treatment C showed a 445.6% relative lift at 99% confidence. This is flagged as unreliable.
- The control converted far below its 3.83% historical rate and had six days in a row with no conversions.
- Only 2 to 10 conversions happened per day, so a few orders swung the rate a lot.
- Rerun with a cloned control: two identical pages differed by 30%, so the original lift could not be trusted.
The change
Treatments reworked the value proposition and page sequence.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
The company's top-selling product page as it was.
Variant
Four treatments that reworked the page's value proposition and the order in which it made its case. Treatment C was the apparent winner.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Too few conversions: daily rates ranged from 0.84% to 5.62%, far from the 15% the team assumed when planning.
- A statistical confidence number can look strong on tiny counts and still be noise.
- A slump in the control, not a better treatment, likely drove most of the gap.
Caveats: Result marked inconclusive: the lift came from very few conversions and failed a cloned-control check. The team switched to a template test across the top 20 products to get roughly ten times more conversions.
Who ran it
MarketingExperiments ran this test for a seller of mostly digital products and services to high-tech industries and published the results (MarketingExperiments). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on MarketingExperiments →
How to run this test on your site
- Hypothesis
- If we rework the value proposition on our product pages, then more visitors will buy because the page will explain more clearly why to choose us.
- Primary metric
- Conversion rate across a product page template
- Guardrails
- A cloned control (A/A check) to measure noise
- Conversion rate versus historical baseline
- Revenue per visitor
- Who should test it
- SaaS and digital product sellers with many similar product pages. Test the shared template, not a single page, unless that page gets plenty of conversions every day.
Questions
Can a test at 99% confidence still be wrong?
Yes. With only a handful of conversions a day, a few orders can create a huge, significant-looking gap that disappears on a rerun.
What is a cloned control?
It is a second copy of the original page run in the same test.
If two identical pages differ a lot, you know how much noise is in your data.
How do you test pages with low traffic?
Group similar pages into one template test so more conversions count toward the result, as this team did with its top 20 products.