Statistical power is the probability that an A/B test detects a real effect of a given size, if that effect exists.
The usual target is 80%: if your variant truly lifts conversion by the amount you planned for, a test with 80% power flags it as significant 8 times out of 10 and misses it 2 times.
Power depends on four inputs: sample size, the effect size you want to detect, the noise in your metric, and your significance threshold.
Power is the third leg of the stats trio, next to statistical significance and the minimum detectable effect.
Significance controls false positives. Power controls false negatives, the Type II errors where a real winner gets labeled "no difference."
Power analysis: what it is and when to run it
A power analysis is the calculation you do before a test starts.
You fix three of the four inputs and solve for the fourth.
In A/B testing you almost always solve for sample size: given my baseline rate, the smallest lift I care about, a 5% significance level and 80% power, how many visitors do I need per variant?
You can also run it the other way.
If you know you can only get 20,000 visitors a month to a page, the analysis tells you the smallest lift you could reliably detect.
That answer often ends a test idea before it wastes a month.
Where the 80% default comes from
The 80% convention comes from psychologist Jacob Cohen.
In Statistical Power Analysis for the Behavioral Sciences (1988), Cohen proposed power of 0.80 as a default, reasoning that a 20% miss rate paired with a 5% false-positive rate treats false positives as about four times as costly as misses.
It is a convention, not a law. If missing a winner is expensive (a pricing change, a checkout rebuild), 90% is a reasonable target.
Pre-test power vs. post-hoc power
Power only means something when you compute it before the test, from an effect size you chose in advance.
"Observed power" computed after the test from the effect you happened to see is a different thing.
Ron Kohavi, Alex Deng and Lukas Vermeer show in A/B Testing Intuition Busters (KDD 2022) that post-hoc power is fully determined by the p-value and the significance threshold, so it adds no new information.
Any result with p above 0.05 has post-hoc power below 50%.
In their example, a test with a p-value of 0.009 showed 74% post-hoc power, while its real pre-experiment power was about 3%.
Keep that in mind when a testing tool prints a "power" number next to a finished result.
Treat it as a restatement of the p-value, not as evidence that the test was well designed.
Power calculation and sample size
For a conversion-rate test with two equal groups, the standard formula for visitors per variant is:
n = (z_alpha/2 + z_beta)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2
Here p1 is the control's conversion rate, p2 is the rate you want to be able to detect, z_alpha/2 is 1.96 for a two-sided 5% test, and z_beta is 0.84 for 80% power (1.28 for 90%).
Kohavi, Deng and Vermeer give a shortcut for 80% power and a 0.05 threshold, citing van Belle: n = 16 x variance / delta^2, where variance for a conversion rate is p(1 - p) and delta is the absolute change you want to detect.
Worked example
Say your baseline conversion rate is 3.0% and you want to detect a 10% relative lift, to 3.3%. The absolute difference is 0.3 percentage points.
- Variance term: 0.03 x 0.97 + 0.033 x 0.967 = 0.0610
- (1.96 + 0.84)^2 = 7.85
- n = 7.85 x 0.0610 / 0.003^2 = about 53,200 visitors per variant
So you need roughly 106,000 visitors across both arms.
The 16-variance shortcut gives about 51,700 per variant, close enough for planning.
Our sample size calculator does this math for you.
How power changes with sample size
Here is the same test (3.0% baseline, true lift to 3.3%, two-sided 5% test) at different sample sizes.
These are our own calculations from the formula above.
| Visitors per variant | Power to detect the 10% lift |
|---|---|
| 2,000 | 8% |
| 5,000 | 14% |
| 10,000 | 23% |
| 25,000 | 48% |
| 53,000 | 80% |
| 70,000 | 89% |
At 10,000 visitors per arm, a real 10% lift is missed more than three times out of four.
Most "inconclusive" tests on smaller stores sit in this zone.
The effect size drives everything
Sample size scales with one over the squared difference.
Halve the lift you want to detect and you need about four times the traffic.
| Relative lift to detect (from 3.0%) | Visitors per variant at 80% power |
|---|---|
| 20% (to 3.6%) | about 13,900 |
| 10% (to 3.3%) | about 53,200 |
| 5% (to 3.15%) | about 207,900 |
This is why a store with 30,000 monthly visitors can test bold changes but not button tweaks.
See our guide to testing with low traffic for what to do instead.
Other levers besides traffic
- Pick a less noisy metric. Add-to-cart rate has more events than purchases, so it reaches the same power with fewer visitors. Revenue per visitor is noisier than conversion rate.
- Reduce variance. Methods like CUPED use pre-experiment data to cut metric variance, which raises power at the same sample size.
- Test on the page with the traffic. A sitewide header change gets far more exposure than a single product page.
- Run fewer variants. Each extra arm splits the traffic further.
Once you have the number, translate it into calendar time with how long to run an A/B test, and run full weeks.
Statistical power in A/B testing: what real tests show
Our database of real A/B test results includes tests where the team reported sample size and power.
They show both sides of the problem: underpowered tests that produce big, unreliable numbers, and well-powered tests that produce trustworthy nulls.
Underpowered tests produce big numbers
Clean Commit publishes a power figure with each test.
Their test sending ad traffic to the homepage showed +103.9% conversion rate and +137.6% revenue per visitor on 2,777 visitors, with 7% reported power.
The agency labeled it a positive signal, not a winner.
Their shorter mobile product gallery showed +24.0% conversion rate on 4,788 visitors with 95.3% probability to beat control, but only 23% power, and was counted as not significant.
A free-shipping line on product pages showed +21.0% conversion rate on 4,128 visitors over 26 days, with 15% power.
This pattern is expected. Gelman and Carlin's paper Beyond Power Calculations (2014) shows that when power is low, the estimates that do reach significance are exaggerated, and some point in the wrong direction.
They call these Type M (magnitude) and Type S (sign) errors.
Kohavi, Deng and Vermeer note that below 50% power the exaggeration ratio becomes so high as to be meaningless. That is the winner's curse in action.
Early peeks look like winners
Ruben ran an authority badge test that showed +17% conversion rate after one week at 99.6% probability to beat control.
They kept it running. After four weeks the lift was +1.66%, flat and most likely noise.
A one-week sample was too small to estimate the effect, even though the probability looked decisive.
Fixing the sample size in advance, or using a proper sequential test, protects you from this.
Well-powered nulls are real answers
With enough traffic, a flat result means something.
Clean Commit's trust icons at checkout ran on 213,715 visitors for 40 days: +0.9% conversion rate, which the agency called a well-powered null.
GrowthRock's "You Save #%" test ran 280,000 visitors per variation for two weeks, and all three versions landed within 0.3% of each other.
Dave Diederen's "Clinician Recommended" badge was flat on conversion rate, revenue per visitor and AOV across 76k+ visitors.
These tests let the teams drop an idea with confidence.
An underpowered flat result can't do that; it only tells you the test was too small.
What a planned test looks like
Digispot's simplified product page for Mamabrum is a good model.
The team planned for 80% power and a 5.65% minimum detectable effect, then ran it for 5 weeks on 148,710 visitors with a 50/50 split.
Conversion went from 3.63% to 3.85% (+6.03%), and add-to-cart rose 11.64%.
Because the observed lift was close to the planned MDE, the result is believable in a way a +60% result on 3,000 visitors is not.
Clean CommitClean CommitClean Commit
GrowthRock
@RubendgrfWhen a standard power analysis is the wrong tool
- You plan to peek and stop early. The fixed-sample formula assumes one look at the end. If you check daily, use a sequential design with its own sizing.
- Your metric is heavily skewed. Revenue per visitor has outliers, so the p(1 - p) variance doesn't apply. Use the metric's historical variance instead.
- You run many variants or metrics. Correcting for multiple comparisons lowers the effective significance threshold, which raises the sample you need.
- You're judging a finished test. Don't compute power from the observed effect. Look at the confidence interval instead: a wide interval that includes both zero and meaningful lifts means the test was too small.
For the full workflow around these numbers, start at our A/B testing hub.