Ad traffic sent to the homepage: +103.9% conversion (tiny sample)
Test run byClean CommitCRO agency
Commentary byLiftArchiveOur analysis, not theirsSending visitors bound for a sport-specific landing page to the homepage instead more than doubled conversion rate (+103.9%), but the sample was too small to trust, so we count it as inconclusive.
Clean Commit ran this test for an anonymized Shopify store and published it in its A/B test library.
The test reached only 2,777 visitors with 7% statistical power, and was run as a stopgap while a new landing page was built.
- Add-to-cart rate: +83.5%, the main metric, with an 81.2% probability of beating the original. Not significant.
- Conversion rate: +103.9%, and revenue per visitor: +137.6%, both on only 2,777 visitors.
- Click-through rate: -18.0% and bounce rate: +5.7%, so fewer visitors explored further.
- Statistical power was 7%, far too low to treat these gains as proof.
The change
Visitors land on the full homepage instead of a sport-specific page.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
Traffic went to a sport-specific landing page.
Variant
Most of that traffic went to the homepage instead, with a smaller share kept on the original landing page as the control.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- The homepage offers broader navigation and newer sections that may suit visitors better than a narrow sport page.
- The old landing page may simply have been weak, so almost any alternative could look better.
- With so few visitors, a handful of extra orders can create a huge percentage swing.
Caveats: Only 2,777 visitors with 7% power, and the control got a smaller share of traffic, so the size of the lift is unreliable. It was a stopgap test while a new landing page was built, and duration was not shared.
Who ran it
Clean Commit ran this test for a Shopify store (anonymized) and published the results (Clean Commit A/B test library). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Clean Commit →
How to run this test on your store
- Hypothesis
- If we send landing page traffic to the homepage, then add-to-cart rate will rise because the homepage offers broader navigation that better matches what visitors want.
- Primary metric
- Add-to-cart rate
- Guardrails
- Bounce rate
- Revenue per visitor
- Conversion rate
- Who should test it
- Stores with an old or weak campaign landing page and no time to rebuild it. Plan enough traffic for a powered test, and expect a strong, well-matched landing page to beat the homepage.
Questions
Should ad traffic go to the homepage or a landing page?
A well-matched landing page usually wins, but a weak one may lose to the homepage.
This test hinted the homepage did better, on far too small a sample to be sure.
What does 7% statistical power mean?
It means the test had little chance of detecting a real effect reliably.
Large swings on small samples like this often shrink or vanish when retested.
Similar tests
@DaveDiederen
Convertibles@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen