Variance is the average squared distance of each value from the mean, and standard deviation is its square root, which puts the spread back in the original units. In A/B testing, variance tells you how noisy a metric is: the higher it is, the more visitors you need before a difference between control and variant means anything.
The formulas
For a sample of N values, the NIST/SEMATECH e-Handbook of Statistical Methods defines sample variance with an N - 1 denominator:
| Measure | Formula | Units |
|---|---|---|
| Sample variance | s² = Σ(Yi - Ȳ)² / (N - 1) | Squared (dollars², for example) |
| Standard deviation | s = √s² | Same as the data |
| Conversion (yes/no) metric | variance = p(1 - p) | Per visitor |
| Standard error of a mean or rate | SE = s / √n | Same as the data |
The p(1 - p) line follows from the binomial distribution, whose standard deviation is √(np(1 - p)), so its variance is np(1 - p).
Divide that variance by n for a single visitor.
Standard deviation in an A/B test
Your testing tool never shows you a raw standard deviation, but it uses one for every significance result and every confidence interval.
The standard error, s divided by the square root of the sample, is what shrinks as traffic grows.
That is why variance drives sample size. Required sample scales with variance divided by the squared effect you want to detect, so doubling the variance roughly doubles the visitors you need.
Methods like CUPED exist to cut variance so tests finish sooner.
Worked example with a real test
Convertibles tested a full cart drawer against an add-to-cart banner for a high-ticket equipment store.
Conversion went from 0.382% to 0.444% (+16%) over 84,005 visitors and 347 orders, with an 86-92% probability to beat control.
Our calculation, assuming an even split of about 42,000 visitors per arm:
- Control variance: 0.00382 × 0.99618 = 0.0038, so the standard deviation is 0.0617 per visitor.
- Control standard error: 0.0617 / √42,002 = 0.00030, or 0.030 percentage points.
- Variant standard error: 0.0665 / √42,002 = 0.00032.
- Standard error of the difference: √(0.00030² + 0.00032²) = 0.00044, or 0.044 points.
- Observed difference: 0.062 points, which is about 1.4 standard errors.
A 1.4 standard-error gap is short of the roughly 1.96 needed for 95% two-sided significance, which matches the source's own caveat that the result sat below 95% confidence.
A 16% lift on a 0.4% baseline is still small next to the noise.
Why revenue metrics are noisier
Revenue per visitor and average order value have much higher variance than conversion rate, because a few large orders sit far from the mean.
NIST notes that squaring distances gives far more weight to extreme values, so variance "can be greatly affected by the tail behavior."
The database shows it. Clean Commit's collection-page filters test logged -6.4% conversion but -33.2% revenue per visitor and -29.2% AOV, with the revenue drop driven by mobile AOV and possibly a few large orders.
Its same-category recommendations test showed -9.9% conversion yet +19.7% AOV, again with the desktop AOV jump likely from a few large orders.
Anatta handled this up front in its tabbed buy box test: wholesale orders, outliers and orders under $20 were filtered out before reporting AOV going from $102.63 to $104.74 (about +2%) with 97.6% probability of a positive effect.
Convertibles
Clean CommitClean CommitWhen standard deviation is the wrong summary
For heavily skewed data, such as order values with a long tail, a single standard deviation describes the spread poorly.
NIST suggests the median absolute deviation or interquartile range for data with extreme tails.
In testing, the usual fixes are capping outliers at a set percentile or excluding order types like wholesale, decided before the test starts.
Related terms
- Sample size: how many visitors you need, driven by variance.
- Statistical power: the deeper guide on detecting real effects.
- Browse the full glossary or real results in the test database.