Most store A/B tests need to run for at least two full weeks, and often four to eight. The real answer depends on your sample size: work out how many visitors each variant needs, divide by your daily traffic to the tested page, then round up to whole weeks. Stop when you hit that number, not when the dashboard first shows a winner.
Below is the method, a worked example, the rules for stopping early, and what to do when your store doesn't have enough traffic for any of this.
Start with sample size, then convert it to days
Duration is an output, not a setting. You can't pick "two weeks" up front and hope it's enough. First decide what you need to detect, then let the math tell you how long that takes.
You need four inputs:
- Baseline conversion rate of the page or step you are testing (not your sitewide rate).
- Minimum detectable effect (MDE): the smallest lift worth acting on. See our guide to choosing a minimum detectable effect.
- Significance and power: most tools default to 95% confidence and 80% power.
- Daily visitors who actually reach the tested page and enter the test.
Our A/B test sample size calculator does the math for you. If you want to check it by hand, Evan Miller gives a rule of thumb in How Not to Run an A/B Test: n = 16σ²/δ² visitors per variant, where σ² = p(1 − p) for a conversion rate p, and δ is the absolute difference you want to detect.
Worked example
Say your product page converts at 3% and you want to detect a 10% relative lift (3.0% to 3.3%, so δ = 0.003).
- σ² = 0.03 × 0.97 = 0.0291
- n = 16 × 0.0291 ÷ 0.000009 ≈ 51,700 visitors per variant
- Two variants ≈ 103,500 visitors in total
With 2,000 visitors a day on that page, that's about 52 days. Round up to full weeks and you get 8 weeks.
Now say you only care about a 20% relative lift (3.0% to 3.6%). Doubling δ cuts the sample by four: about 12,900 per variant, or 25,900 in total. At 2,000 a day that's 13 days, which you round up to 14.
That's the main lever. Small effects need huge samples. Halving the effect you want to detect makes the test four times longer.
One trap: use traffic for the exact page or flow you're testing. Convert's test duration calculator notes that sitewide traffic "makes the duration look artificially short." A checkout test only counts people who reach checkout.
Run full weeks and full business cycles
Once the math gives you a number of days, round it up to whole weeks. Shoppers behave differently on Monday morning than on Saturday night, and a test that stops mid-week over-counts some days.
Convert's calculator puts it this way: cover complete business cycles "even when the sample is reached mid-cycle," because traffic and behavior vary across "weekdays, weekends, billing cycles, promotions, and holidays." For most stores the cycle is one week. Two full weeks is a sensible floor, even for high-traffic tests.
Some rules that follow from this:
- Start and stop on the same weekday. A test that runs Monday to Monday covers every day evenly.
- Avoid big promos unless the promo is the test. A Black Friday week sends very different traffic. If a sale lands mid-test, run long enough that it doesn't dominate the data, or restart afterward.
- Watch for paydays and email sends. If you mail your list every other Thursday, a one-week test may catch one send or none.
- Don't run for months. Past roughly six to eight weeks, cookie churn and returning visitors seeing both versions start to muddy results. If the math says three months, the test is too small for your traffic (see the low-traffic section below).
Results also differ from one store to the next, so a winner elsewhere still needs a full run on your own traffic. Dave Diederen saw +3.16% revenue per visitor from a checkout countdown timer on one store, −7.1% RPV from the same idea on another, and −35.61% RPV in a European market. Ruben, at Tailored Commerce, tested a countdown timer across several brands and got about +10% CVR on a low-priced brand, flat on a high-priced one, and −9% CVR on another.
When to stop early (and when not to)
The most common way to get a fake winner is checking results every day and stopping the first time the number turns green. Evan Miller showed that repeated peeking with a standard test can push a nominal 5% false positive rate up to 26.1%. That means about one "win" in four could be noise.
Our database has a clear example. Ruben tested an authority badge (a doctor or expert endorsement) on a product page. After week one it showed +17%. After four weeks it settled at +1.66% CVR, basically flat. Stopping after week one would have shipped a change that did almost nothing.
Want results like this on your store?
Ruben at Tailored Commerce. Ruben runs CRO at Tailored Commerce for brands like Hears, Moov and FLÂNEUR.
Early lifts often fade because the first days over-represent your most engaged returning visitors, and because small samples swing hard. Optimizely's own launch guidance warns that "results may fluctuate and conversions may drop before your test reaches statistical significance," and says to let the test run until the time to completion you planned.
Valid reasons to stop early
- Something is broken. A variant that throws errors, breaks checkout, or tanks revenue badly should be paused. That's a QA problem, not a result.
- You're using a method built for continuous monitoring. Sequential testing adjusts its thresholds so you can look while the test runs. Optimizely describes its Stats Engine as giving results that stay valid when you check them. Even then, finish at least one full week.
- You pre-planned checkpoints. A sequential design with set interim looks and stricter thresholds at each look is fine. Making up stopping rules halfway through is not.
Bad reasons to stop
- The tool shows 95% significance on day three.
- The variant "looks like it's losing" before the sample is in.
- You need the slot for the next test.
Also know what your tool's number means. Kasper Dolk at Maison Commerce reports results such as +€127,112 revenue lift at 90% confidence for a new navigation, and a collection-page quiz that was at +€54,910 revenue lift, 89% confidence, while still in progress. A 90% threshold is a business choice that accepts more risk than 95%. Pick your threshold before launch and write it down. Our guide to statistical significance explains the trade-offs.
Want results like this on your store?
Kasper Dolk at Maison Commerce. Kasper runs Maison Commerce, which builds CRO programs, landing pages and themes for fast-growing ecommerce brands such as ICON and HiStrips.
Picking a tool for this
Your testing tool decides which stopping rules you can use. Convert offers frequentist, Bayesian and sequential engines. As of November 2026, Convert's pricing page lists the Growth plan at $299/month billed annually ($399 monthly) and Pro at $420/month billed annually ($599 monthly), both for 100K tested users a month, with a 15-day free trial. The comparison tables show sequential testing as standard on Pro and Enterprise. That price is a lot for a small store, so it's not the right fit if you're testing on a few thousand visitors a month. Read our Convert review for the full picture.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
What to do on a low-traffic store
If the calculator says your test needs 12 weeks or more, don't run it as planned. Change the test instead. You have four options.
1. Test bigger changes. A larger expected effect shrinks the sample a lot. Tweaking a button color won't move enough to measure on small traffic. Rebuilding a page might. Dave Diederen saw +52.67% RPV from replacing a homepage hero video with a static image and benefit callout, and Milos saw +128% RPV (+$14,867/month) from rebuilding a best-seller product page. Lifts that size show up in weeks, not months.
2. Test where the traffic is. Your homepage, top product pages and cart drawer see far more visitors than a niche collection page. Put your first tests there.
3. Measure a metric closer to the change. Add-to-cart happens far more often than purchase, so a test on add-to-cart rate needs fewer visitors. The catch is that more add-to-carts don't always mean more orders. See our guide to A/B testing metrics.
4. Accept a higher MDE or a lower confidence level, on purpose. Detecting only 20%+ lifts, or deciding at 90% instead of 95%, shortens the test. Just decide that before launch, not after you see the data.
If none of those fit, skip the split test. Use customer research, session recordings, and changes that worked for similar stores in our test database, then watch your before-and-after numbers. Our low-traffic A/B testing guide covers this in more depth.