Sample size is the number of people or observations you measure in a study or experiment. In A/B testing, it is how many visitors (or users, or sessions) each version needs to see before the result can reliably tell a real difference from random noise.
Pick it before the test starts, not after.
A sample that is too small leaves a test unable to detect anything but very large effects, and it makes the "winners" it does find less trustworthy.
Sample size definition in A/B testing
In statistics, a sample is the part of a population you actually observe, and sample size (written n) is how many units are in it.
On a website, the population is all your future visitors.
The sample is everyone who enters the test.
The sample size you need depends on four inputs:
- Baseline rate. Your current baseline conversion rate. Lower baselines need more visitors.
- Minimum detectable effect. The smallest lift you care about finding. See minimum detectable effect.
- Significance level. Usually 5%, meaning 95% confidence.
- Power. The chance of detecting a real effect of that size, usually 80% or 90%. See statistical power.
Noisier metrics also need bigger samples. Revenue per visitor varies far more than a yes/no conversion, so it takes many more visitors to detect the same percentage change.
A worked example
Ron Kohavi and colleagues give a rule of thumb in their 2007 KDD paper on controlled experiments: n = (4rσ / Δ)², where r is the number of variants, σ is the standard deviation of the metric, and Δ is the smallest difference you want to detect.
It assumes 95% confidence and 90% power. For a conversion rate p, σ² = p(1 − p).
The paper's own example: a store converts 5% of visitors and wants to detect a 5% relative change (5.00% to 5.25%).
That needs just under 500,000 users. To detect a 20% change instead, the number drops by a factor of 16, to about 30,400.
| Baseline conversion | Change to detect | Users needed (Kohavi et al.) |
|---|---|---|
| 5% | 5% relative (0.25 points) | Just under 500,000 |
| 5% | 20% relative (1 point) | About 30,400 |
That 16x drop is the key point. Sample size grows with the square of the effect you want to see, so halving the minimum detectable effect roughly quadruples the visitors you need.
The paper also notes the factor of 4 can overestimate by about 25% at large n.
A sample size calculator gives tighter numbers for your own baseline.
What our database shows
Tests that planned their sample size read very differently from tests that didn't.
Digispot planned a simplified product page test for Mamabrum at 80% power for a 5.65% minimum detectable effect.
It ran 5 weeks on 148,710 visitors, split 50/50, and purchase conversion went from 3.63% to 3.85% (+6.03%).
Clean Commit ran a checkout trust icons test for 40 days on 213,715 visitors and reported 100% power.
The result was +0.9%, which they called no clear effect.
With a sample that large, a flat result is real information: the change didn't do much.
Small samples swing wildly. Clean Commit's "Made in the USA" icon test showed +53.0% conversion on 1,408 visitors over 38 days, and the agency still labeled it directional, not proven.
Server Density's packaged pricing test reported +114% revenue from just 20 vs 15 paid conversions.
Clean CommitClean CommitWhen sample size thinking goes wrong
- Stopping when it looks good. The sample size is a target to reach, not a ceiling to check against every day. Stopping at the first significant reading inflates false positives.
- Counting visitors, ignoring conversions. A test with 20,000 visitors but 35 conversions is still a small test.
- Counting the wrong visitors. If only checkout starters can see a change, Kohavi et al. recommend analyzing only those users, since everyone else adds noise.
Related terms
Sample size sits alongside the baseline conversion rate, minimum detectable effect and statistical power above. For more terms, browse the CRO glossary.