Homepage tailored to new vs returning visitors: +241% CR, unproven
Test run byClean CommitCRO agency
Commentary byLiftArchiveOur analysis, not theirsShowing new visitors an educational homepage and returning visitors a sales-focused one came with a 241.1% rise in conversion rate, but the test did not reach significance.
Clean Commit ran this test for an anonymized Shopify store and labelled it a positive signal.
Overlapping tests and holiday promotions interfered, so we count it as inconclusive.
- Conversion rate: +241.1%. Very large, but not statistically significant.
- Add-to-cart rate: +83.7%, and average order value: +106.0%.
- Revenue per visitor: +602.8% (goal metric). Numbers this large usually point to noise or outside interference.
- The agency reported that new visitors spent more time on the page, but gave no figure.
The change
The variant shows offers to returning visitors and educational, trust content to new ones.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
All visitors saw the same homepage.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Returning visitors often come back to buy, so a shorter path with offers may suit them.
- New visitors may need answers and reassurance before buying, which the educational layout aimed to give.
- Holiday promotions and overlapping tests likely inflated the numbers, so the true effect is unknown.
Caveats: The test did not reach significance, and other tests and holiday promotions ran at the same time. Several layout and content changes for two segments were tested together, and sample size and duration were not shared.
Who ran it
Clean Commit ran this test for a Shopify store (anonymized) and published the results (Clean Commit A/B test library). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Clean Commit →
How to run this test on your store
- Hypothesis
- If we show new visitors educational, trust-building content and returning visitors offers and a quick path to buy, then conversion rate will rise in both groups because each gets what they came for.
- Primary metric
- Revenue per visitor
- Guardrails
- Conversion rate by segment
- Bounce rate for new visitors
- Average order value
- Who should test it
- Stores with a healthy share of repeat visitors and a product that needs some explaining. Run it outside big promotions, with no overlapping tests on the homepage.
Questions
Does homepage personalization for new and returning visitors work?
It may, but this test can't prove it.
The lifts were very large but not significant, and holiday promotions and other tests interfered.
What should new visitors see on a homepage?
Content that explains the product and builds trust, such as how it works, reviews and answers to common questions.
Returning visitors usually want a faster way to buy.
Why are huge test lifts often unreliable?
Very large changes usually come from small samples, outside events like sales, or overlapping tests.
Rerun the test in a quiet period before trusting it.
Similar tests
@oliverkenyon
@DaveDiederen
Convertibles@DaveDiederen@DaveDiederen@DaveDiederen