An A/B test needs enough visitors per variant to spot your minimum detectable effect at your chosen confidence and power. For a typical store with a 3% conversion rate, finding a 10% relative lift at 95% confidence and 80% power takes about 53,200 visitors per variant, or roughly 106,400 in total. Enter your own numbers in the calculator above, and the steps below explain where the figure comes from.
What the calculator inputs mean
The calculator takes four inputs. These are the same ones used by standard tools like Evan Miller's sample size calculator.
- Baseline conversion rate: the rate your control page gets today. Pull it from the same page and the same audience you plan to test, not your sitewide average. Our conversion rate calculator can help, and the ecommerce conversion rate benchmarks show typical ranges.
- Minimum detectable effect (MDE): the smallest relative lift you care about finding. A 10% MDE on a 3% baseline means you want to detect a move to 3.3%. Our guide to choosing a minimum detectable effect covers this in more depth.
- Confidence (1 − alpha): how often you can accept a false positive. 95% means a 5% chance of calling a winner when there is no real difference.
- Power (1 − beta): how often the test catches a real effect of your MDE size. 80% is the usual default, and it means one in five real winners of that size will still look flat.
How to read the output
The main number is visitors per variant. Multiply it by the number of variants, control included, to get the total traffic you need. An A/B/C test needs 50% more traffic than a plain A/B test.
Treat the number as a stopping rule you set before launch. As Evan Miller explains in How Not To Run an A/B Test, the significance math assumes "that the sample size was fixed in advance." If you check every day and stop as soon as a result looks significant, you end up with far more false winners than your confidence level suggests.
Our database has a clear example. Ruben tested a doctor endorsement badge that showed +17% after the first week. After four weeks it settled at +1.66% CVR, which is basically flat. The early spike was noise.
Want results like this on your store?
Ruben at Tailored Commerce. Ruben runs CRO at Tailored Commerce for brands like Hears, Moov and FLÂNEUR.
The output also tells you which tests are realistic for you. Small MDEs get expensive fast:
| Baseline CVR | Relative MDE | Visitors per variant (95% confidence, 80% power) |
|---|---|---|
| 3% | 20% | 13,914 |
| 3% | 10% | 53,211 |
| 3% | 5% | 207,938 |
| 1% | 10% | 163,095 |
Cutting the MDE in half roughly quadruples the sample. A lower baseline also pushes the sample up, so a page converting at 1% needs about three times the traffic of one converting at 3% for the same relative lift.
This is why many real lifts are hard to confirm. Dave Diederen saw +3.16% revenue per visitor from a checkout countdown timer. On a conversion metric with a 3% baseline, a 3.16% relative lift would need about 516,000 visitors per variant to detect at 80% power. Revenue metrics vary more than conversion rates, so they usually need even more. Only high-traffic stores can confirm effects that small.
The A/B testing sample size formula
For two conversion rates, the standard formula for visitors per variant is:
n = (zα/2 × √(2 × p̄ × (1 − p̄)) + zβ × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)²
- p1 is the baseline rate and p2 = p1 × (1 + MDE).
- p̄ is the average of p1 and p2.
- zα/2 is 1.96 for 95% confidence (two-sided).
- zβ is 0.84 for 80% power, or 1.28 for 90% power.
Worked example
Take a baseline of 3% and an MDE of 10%. Then p1 = 0.03, p2 = 0.033 and p̄ = 0.0315. The difference you are trying to detect is 0.003, or 0.3 percentage points.
- First term: 1.96 × √(2 × 0.0315 × 0.9685) = 1.96 × 0.2470 ≈ 0.4841.
- Second term: 0.8416 × √(0.0291 + 0.0319) = 0.8416 × 0.2470 ≈ 0.2079.
- Add them and square: (0.6920)² ≈ 0.4789.
- Divide by 0.003² = 0.000009: about 53,200 visitors per variant.
Raising power to 90% lifts this to about 71,200 per variant. For a quick sanity check, use Lehr's rule of thumb: n ≈ 16 × p(1 − p) ÷ δ² at 95% confidence and 80% power, as described in this paper on early-stage experimentation. With our numbers, 16 × 0.03 × 0.97 ÷ 0.000009 ≈ 51,700, which is close.
The formula also shows what a real result needs. Elevate's pink vs green add-to-cart button test found 11.49% vs 10.81% add-to-cart, a 6.3% relative lift. To detect a lift of that size reliably from a 10.81% baseline, you would need about 33,600 visitors per variant.
When the test is over, check the result with our A/B test significance calculator. For background on p-values, see statistical significance explained.
Days needed at your traffic
To turn visitors into days, divide the total sample by the daily visitors who actually reach the tested page:
Days = (visitors per variant × number of variants) ÷ daily visitors to the tested page
With the worked example above (53,211 per variant, two variants):
- At 5,000 visitors a day: 106,422 ÷ 5,000 ≈ 22 days. Round up to 28 days so the test covers four full weeks.
- At 1,500 visitors a day: about 71 days. That is too long for most stores, so raise the MDE, test a page with more traffic, or use a metric that converts more often, like add-to-cart.
Always run in full weeks, because weekday and weekend shoppers behave differently. Our guide on how long to run an A/B test covers this, and the low-traffic testing guide has options for smaller stores.
Results also change by audience, so a sample from one store does not carry over to another. Ruben ran a countdown timer across several brands and got about +10% CVR on a low-priced brand, a flat result on a high-priced brand and -9% CVR on another. Kasper Dolk's product finder quiz was reported at +€54,910 revenue lift with 89% confidence while still in progress, which shows why the stopping point should be planned before launch.
Once you know your sample size, you need a tool that splits traffic evenly and reports results per variant. Compare the options in our A/B testing tools roundup, or browse all 285 tests in the database for ideas that are worth the traffic.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Also works well
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
Questions
How do I calculate sample size for an A/B test?
Pick your baseline conversion rate, the smallest relative lift you want to detect (MDE), a confidence level (usually 95%) and power (usually 80%), then use the two-proportion formula or the calculator above. At a 3% baseline and a 10% MDE, you need about 53,200 visitors per variant.
What is the A/B testing sample size formula?
n = (z(α/2) × √(2p̄(1 − p̄)) + z(β) × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)², where p1 is the baseline, p2 is the expected rate and p̄ is their average. A quick approximation at 95% confidence and 80% power is n ≈ 16 × p(1 − p) ÷ δ².
What is a minimum detectable effect?
It is the smallest lift your test is set up to detect reliably. A 10% relative MDE on a 3% conversion rate means detecting a change to 3.3%. Halving the MDE roughly quadruples the visitors you need.
What power should I use for an A/B test?
80% is the common default. It means a real effect of your MDE size is detected four times out of five. Moving to 90% power raises the 3% baseline, 10% MDE example from about 53,200 to about 71,200 visitors per variant.
How many days should my A/B test run?
Divide the total sample (visitors per variant times number of variants) by the daily visitors to the tested page, then round up to full weeks. For example, 106,422 visitors at 5,000 a day is about 22 days, so run it for 28.