Homepage personalized for new vs returning visitors: flat result
Test run byClean CommitCRO agency
Commentary byLiftArchiveOur analysis, not theirsShowing new visitors an educational homepage and returning visitors a sales-focused one changed conversion rate by just 0.5%, with revenue per visitor up 6.7% on weak odds.
Clean Commit ran this test for an anonymized Shopify store and published it in its A/B test library.
Mobile and desktop moved in opposite directions, so we count it as inconclusive.
- Conversion rate: +0.5%. No clear difference.
- Revenue per visitor (the goal metric): +6.7%, with only a 58.6% probability of beating the original.
- Add-to-cart rate: +11.6% and click-through rate +16.7%. More browsing, not clearly more buying.
- By device: conversion rate +17.7% on mobile but -24.4% on desktop. The two cancelled out.
The change
The variant serves different homepage content to new and returning visitors.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
Every visitor saw the same homepage.
Variant
New visitors saw a homepage that introduced the brand and explained its main benefits. Returning visitors saw product highlights, offers and direct calls to action.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Educational content may help a first-time mobile visitor but slow down a desktop visitor who came ready to shop.
- Engagement metrics rose, which suggests the pages got attention without changing the decision to buy.
- Splitting traffic into two experiences halves the data behind each one, which makes a clear read harder.
Caveats: The goal metric had a 58.6% probability of beating the original, which is close to a coin flip. Sample size and duration were not shared.
Who ran it
Clean Commit ran this test for a Shopify store (anonymized) and published the results (Clean Commit A/B test library). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Clean Commit →
How to run this test on your store
- Hypothesis
- If we show new visitors an educational homepage and returning visitors a shop-first homepage, then revenue per visitor will rise because each group gets what it needs at its stage.
- Primary metric
- Revenue per visitor
- Guardrails
- Conversion rate by device
- Conversion rate for new vs returning visitors
- Who should test it
- Brands with a product that needs explaining and a healthy share of repeat visitors. Plan to read results by device and visitor type, and expect to need a lot of traffic.
Questions
Does homepage personalization increase sales?
Not clearly in this test. Conversion rate barely moved overall, and mobile and desktop reacted in opposite ways.
Why check results by device?
A change can help one device and hurt another.
Here mobile rose 17.7% while desktop fell 24.4%, which hid both effects in the total.
Similar tests
@DaveDiederen
Convertibles@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen