A statistical power calculator tells you the chance your A/B test will detect a real effect of a given size, or, flipped around, the smallest effect (the minimum detectable effect, or MDE) your traffic can reliably catch.
Enter your baseline conversion rate, visitors per variant and significance level, and it returns power for a target lift, or the MDE at 80% power.
What the MDE calculator works out
Power is the probability that a test reaches significance when a true effect of a stated size exists.
A test with 80% power misses a real effect of that size one time in five.
The 80% convention traces back to Jacob Cohen, who treated a missed effect as about a quarter as costly as a false alarm (summary of Cohen's reasoning).
The calculator above answers two linked questions. With your traffic fixed, what MDE can you detect?
With a target lift fixed, what power do you have?
If you instead want to know how many visitors a given lift needs, use the A/B test sample size calculator.
Power calculator for an A/B test: inputs and formula
The calculator uses the standard normal approximation for comparing two conversion rates with equal traffic in each arm.
| Input | What it means | Typical value |
|---|---|---|
| Baseline rate (p1) | Control conversion rate | From your analytics, last 4-8 weeks |
| Visitors per variant (n) | Traffic each arm will get over the planned run | Weekly traffic x weeks / number of arms |
| Significance level (alpha) | False positive rate you accept | 0.05, two-sided |
| Target power (1 - beta) | Chance of catching a real effect of size MDE | 80% |
Power for a relative lift, where p2 = p1 x (1 + lift):
SE = sqrt( [p1(1-p1) + p2(1-p2)] / n )
Power = Phi( |p2 - p1| / SE - z(1 - alpha/2) )
Phi is the standard normal cumulative distribution and z(0.975) = 1.96. The MDE is the lift where power equals your target, which means the absolute difference must be about (1.96 + 0.84) x SE for 80% power. The calculator solves that for you.
A useful shortcut: at 95% confidence and 80% power, each arm needs roughly 16 x p(1-p) / d^2 visitors, where d is the absolute difference, since 2 x (1.96 + 0.84)^2 is about 15.7. Ron Kohavi and colleagues give the same style of rule for 90% power, and their worked example shows that detecting a 5% change in revenue per user can take over 1.6 million users because revenue is so noisy (Kohavi, Henne and Sommerfield, KDD 2007).
Minimum detectable effect calculator: a worked example
Say a store converts at 3% and can send 10,000 visitors to each of two variants.
These numbers are our calculation with the formula above, two-sided alpha 0.05.
| Visitors per variant | MDE at 80% power (relative) | Power to detect a +10% lift |
|---|---|---|
| 5,000 | 34.4% | 13.5% |
| 10,000 | 23.8% | 22.8% |
| 20,000 | 16.6% | 40.4% |
| 40,000 | 11.6% | 68.1% |
| 80,000 | 8.1% | 93.0% |
At 10,000 per arm, the test can only reliably catch a lift of about 24%, taking 3.0% to roughly 3.7%.
A real +10% lift would be missed more than three times out of four.
Reaching 80% power on +10% takes about 53,000 visitors per arm.
Notice that halving the MDE roughly quadruples the traffic you need.
That square law is why low-traffic sites should test bigger, bolder changes.
How to read the result
- MDE above the lifts you expect: the test is underpowered. A flat result will tell you little. Run longer, test a bigger change, or pick a metric closer to the change, such as add-to-cart instead of revenue.
- Power of 80% or more at a plausible lift: go ahead. If the result is flat, you can say with some confidence that any effect is smaller than your MDE.
- Power below 50%: a "win" from this test is more likely to be noise or an inflated estimate. Kohavi and coauthors show the false positive risk climbs sharply as power drops (A/B Testing Intuition Busters); see statistical power for those figures.
What real tests in our database show
Clean Commit publishes power next to each result, which makes their library a useful reality check.
A free-shipping line on product pages showed +21.0% conversion on 4,128 visitors over 26 days, but with 15% power it was labeled a positive signal, not a win.
A condensed product gallery reported +24.0% conversion and 95.3% probability to beat control on 4,788 visitors at 23% power, and still counts as inconclusive.
Compare trust icons added to checkout: 213,715 visitors over 40 days at 100% power, with +0.9% conversion.
That flat result carries weight because the test could have caught a meaningful effect.
The free-shipping message moved below add to cart, at 76,500 visitors and 99% power, came back at -0.2% conversion.
Digispot planned its Mamabrum product page redesign for 80% power and a 5.65% MDE, then ran 148,710 visitors over 5 weeks and saw conversion move from 3.63% to 3.85% (+6.03%).
The observed lift landed just above the planned MDE, which is what a well-sized test looks like.
Clean CommitClean CommitClean CommitCommon mistakes
- Calculating power after the test from the observed lift. This "observed power" is a direct function of the p-value, so a non-significant result always shows low observed power. It adds no information (Hoenig and Heisey, 2001). Do the calculation before launch, with the smallest lift worth acting on.
- Setting the MDE to the lift you hope for. Set it to the smallest lift that would pay for the change. Hoped-for lifts are usually too large, which hides underpowered tests.
- Not matching settings across tools. One-sided tests, 90% confidence or uneven splits all change the answer. With two-sided 95% our formula gives the Mamabrum test about 55% power at 5.65%, so Digispot's plan likely used different settings. Check your tail choice before comparing numbers.
- Forgetting extra variants. Each extra arm splits traffic and adds multiple comparisons, so enter visitors per variant, not total.
- Stopping when the result looks good. Power assumes you analyze once at the planned sample. Peeking inflates false positives; see how long to run an A/B test.
For more on choosing the effect size itself, read our guide to the minimum detectable effect, then browse all tests to see what lifts similar changes produced.