Promo pop-up delayed by 60 seconds: +7.9% conversion, inconclusive
Test run byClean CommitCRO agency
Commentary byLiftArchiveOur analysis, not theirsDelaying a promo pop-up until shoppers had spent 60 seconds on the page lifted conversion rate by 7.9%, but the result was not significant, so we count it as inconclusive.
Clean Commit ran this test for an anonymized Shopify store and published it in its A/B test library. Mobile improved while desktop got worse.
- Conversion rate: +7.9%, with only a 66.6% probability of beating the original. Not significant.
- Add-to-cart rate: +14.3%, and orders rose 10.2%.
- Revenue per visitor: +4.8%, held back by a 4.8% fall in average order value.
- By device: mobile conversion rose 15.0%, desktop conversion fell 10.1%.
The change
The variant holds the promo pop-up back until 60 seconds on the page.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
A promotional pop-up that appeared as soon as a visitor arrived on the page.
Variant
The same pop-up, shown only after a visitor had spent 60 seconds on the page.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Without an instant interruption, shoppers can start browsing and build interest before seeing the offer.
- On mobile, a pop-up on arrival covers most of the screen, so a delay may matter more there.
- On desktop, fewer visitors may stay 60 seconds, so some may have left without seeing the offer.
Caveats: Based on 23,328 visitors, the test had 24% statistical power and did not reach significance. Treat it as an early signal, and note the opposite results on mobile and desktop.
Who ran it
Clean Commit ran this test for a Shopify store (anonymized) and published the results (Clean Commit A/B test library). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Clean Commit →
How to run this test on your store
- Hypothesis
- If we delay the promo pop-up until a visitor has spent 60 seconds on the page, then conversion rate will rise because shoppers build interest before the offer interrupts them.
- Primary metric
- Conversion rate
- Guardrails
- Email sign-up rate from the pop-up
- Desktop conversion rate
- Average order value
- Who should test it
- Stores that show a discount pop-up on arrival, especially with lots of mobile traffic. Consider separate timing for mobile and desktop, and plan enough traffic to reach significance.
Questions
When should a discount pop-up appear?
There is no single best time. In this test, a 60-second delay looked better on mobile and worse on desktop, so test timing by device.
Do pop-ups on arrival hurt conversion?
They can interrupt shoppers before they see a product.
This test hints at that on mobile, but it did not reach significance.
What does low statistical power mean?
It means the test had little chance of detecting a real effect.
At 24% power, a non-significant result tells you little either way.
Similar tests
@oliverkenyon
@DaveDiederen
Surefoot
Conversion Fanatics