A Type I error is a false positive: your A/B test declares a winner when the change did nothing.
A Type II error is a false negative: the test calls the result flat when the change really worked.
You control the first with your significance level (alpha, usually 5%) and the second with statistical power (usually 80%, which means a 20% chance of missing a real effect of the size you planned for).
The terms go back to Jerzy Neyman and Egon Pearson, who wrote about "errors of the first kind" and "errors of the second kind" in their 1933 paper on the most efficient tests of statistical hypotheses.
Their approach was to fix the Type I error rate first, then find the test that keeps the Type II error rate as low as possible.
This page explains both errors in A/B testing terms, shows how to size a test so neither one takes over, and uses real results from our test database to show what each mistake looks like in practice.
False positive vs false negative: the four possible outcomes
Every A/B test starts from a null hypothesis: the variant performs the same as the control.
When the test ends, you either reject that hypothesis (call a winner or loser) or you don't (call it flat).
Reality either matches your call or it doesn't, which gives four outcomes.
| The change really has no effect | The change really has an effect | |
|---|---|---|
| Test says "significant" | Type I error (false positive). Probability = alpha | Correct detection. Probability = power (1 - beta) |
| Test says "not significant" | Correct. Probability = 1 - alpha | Type II error (false negative). Probability = beta |
The cost of each mistake is different. A false positive means you ship a change that does nothing, or something worse, and you credit it with revenue it never earned.
A false negative means you throw away a change that would have made money, and you often stop testing that whole idea.
The two errors trade off against each other.
If you make it harder to call a winner (a lower alpha), you get fewer false positives but more false negatives, unless you add more traffic.
Sample size is the only lever that lowers both at once.
A common rule of thumb comes from the statistician Jacob Cohen, who suggested 80% power alongside a 5% alpha because researchers tend to treat a false positive as about four times as serious as a false negative (summarized by the Effect Size FAQ, citing Cohen 1988).
That ratio is a convention, not a law.
For a checkout change that is expensive to build and hard to roll back, a stricter alpha makes sense.
For a cheap copy tweak, missing a winner may cost you more than shipping a dud.
Type 1 error: calling a winner that isn't there
The Type I error rate is alpha, the threshold you compare the p-value against.
At alpha = 0.05, if you ran many tests of changes that truly did nothing, about 1 in 20 would still come out "significant" through random noise.
That is what statistical significance promises, and no more.
Alpha only holds if you follow the rules the test was designed under.
Three habits push the real false positive rate well above 5%:
- Peeking. Checking results repeatedly and stopping the first time p drops below 0.05. In Evan Miller's simulation of a test checked after every observation (up to 150 observations), the real false positive rate was 26.1%, not 5%. Use a fixed sample size or a proper sequential test if you want to look early.
- Many metrics or many variants. Test five metrics at 5% each and the chance that at least one shows a fake win is about 23% (1 - 0.95^5). See multiple comparisons for corrections.
- Switching to a one-tailed test after the fact. GrowthRock's category link bar on a mobile homepage showed +5% orders over 28 days, 80,000+ visitors and 2,300 conversions per variation. It reached 93% significance one-tailed but was not significant under Optimizely's two-tailed stats, so we count it as inconclusive. Picking the tail before the test is fine. Picking it after you see the data inflates alpha.
Alpha is not the chance your winner is fake
This is the most common misreading. Alpha is the chance of a significant result given the change does nothing.
What you want to know is the reverse: given a significant result, how likely is it that the change does nothing?
Ron Kohavi, Alex Deng and Lukas Vermeer call this the false positive risk in A/B Testing Intuition Busters (KDD 2022).
False positive risk depends on how often your ideas actually work. The formula from the paper is:
FPR = (alpha x pi) / (alpha x pi + power x (1 - pi)), where pi is the share of tested ideas that truly have no effect.
Worked example: you run 100 tests, 10 of the ideas really work, and each test has 80% power.
Counting only winners, the paper uses a one-sided threshold of 0.025. Real winners detected: 10 x 0.80 = 8. False winners: 90 x 0.025 = 2.25. So 2.25 of your 10.25 "winners" are fake, a false positive risk of about 22%, even though alpha looked like 5%.
The paper's table puts real numbers on this:
| Company / source | Success rate | FPR at 80% power | FPR at 50% power | FPR at 20% power |
|---|---|---|---|---|
| Microsoft | 33% | 5.9% | 9.1% | 20.0% |
| Bing | 15% | 15.0% | 22.1% | 41.5% |
| Booking.com, Google Ads, Netflix | 10% | 22.0% | 31.0% | 52.9% |
| Airbnb search | 8% | 26.4% | 36.5% | 59.0% |
Two takeaways. First, the lower your team's true win rate, the more of your winners are noise.
Second, low power doesn't only cause false negatives.
It also makes the winners you do find less trustworthy, and their measured lifts tend to be inflated (the winner's curse).
Win rates in the wild are often low.
Clean Commit publishes its whole test library, and of its 173 tests in our database, 16 won, 35 lost and 122 were flat.
That is roughly a 9% win rate, close to the Booking.com row above.
How to lower Type I errors
- Fix the sample size and the primary metric before launch, and don't stop early unless you use a sequential method.
- Run an A/A test to check that your tool doesn't call fake winners more often than alpha says it should.
- Retest surprising or high-stakes winners. Voices.com removed featured voice actors from its homepage and saw +33% sign-ups in a 10-day, 50/50 test. They reran it to validate and got +28%. A replication like that makes a false positive much less likely than one result on its own.
Type 2 error: missing a winner that was real
A Type II error happens when a real effect exists but your test doesn't have enough data to separate it from noise.
Its probability is beta, and statistical power is 1 - beta.
Power is always tied to a specific effect size: "80% power" means an 80% chance of detecting a lift at least as big as your minimum detectable effect. Smaller real lifts get missed more often.
Worked example: how traffic drives power
Say your conversion rate is 3.0% and the variant really lifts it 10%, to 3.3%.
With a two-sided alpha of 0.05, the standard sample size formula for two proportions is:
n per variant = (z_alpha/2 + z_beta)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2
With z = 1.96 for alpha and 0.84 for 80% power: (2.80)^2 x (0.0291 + 0.0319) / (0.003)^2 = about 53,200 visitors per variant.
Here is what the same real 10% lift looks like at smaller sample sizes (our calculation, normal approximation):
| Visitors per variant | Power to detect 3.0% to 3.3% | Chance of a Type II error |
|---|---|---|
| 2,000 | 8% | 92% |
| 5,000 | 14% | 86% |
| 10,000 | 23% | 77% |
| 20,000 | 40% | 60% |
| 53,000 | 80% | 20% |
If the true lift were 20%, you would need about 13,900 per variant.
If it were 5%, about 208,000. Halving the effect you want to detect roughly quadruples the traffic you need.
Run your own numbers in the sample size calculator.
What underpowered tests look like
Clean Commit reports power on its tests, which makes the problem easy to see:
- A free-shipping line on product pages showed +21.0% conversion rate and +31.7% revenue per visitor, but on 4,128 visitors over 26 days it had 84.6% probability and 15% power. The agency labeled it a positive signal, not a win.
- A shorter mobile product gallery showed +24.0% conversion rate on 4,788 visitors, 95.3% probability to beat control, with 23% power.
- Larger product images in a desktop grid showed +11.9% conversion rate on 2,552 desktop visitors over 16 days, with 70.7% probability and 13% power.
At 13-23% power, a flat result tells you very little.
The change may work and the test simply couldn't tell.
It's also a reminder that big lifts on small samples tend to shrink on retest.
Compare Digispot's simplified product page for Mamabrum. It was planned at 80% power with a 5.65% minimum detectable effect, ran 5 weeks on 148,710 visitors split 50/50, and lifted conversion from 3.63% to 3.85% (+6.03%).
That is a much smaller lift than the Clean Commit numbers, but the test was sized to detect it.
Clean CommitClean CommitClean Commit
GrowthRockA flat test may be a weak execution, not a dead idea
A false negative isn't only statistical. Paperstone tested bulk-buy discounts shown as a yellow sidebar ribbon and saw no change in conversions.
The team blamed banner blindness, redesigned how the offer was shown and retested it with over 10,000 visitors per version: +16.85% revenue and +18.94% average order value.
Had they read the first flat result as "discounts don't work," that would have been a costly false negative about the idea itself.
How to lower Type II errors
- Calculate the sample size before launch from your baseline rate and the smallest lift worth shipping, then run the full duration.
- Test bolder changes. Larger real effects need far less traffic to detect.
- Pick a primary metric close to the change, such as add-to-cart for a product page tweak, which moves more than revenue does.
- Use variance reduction like CUPED if your tool supports it.
- On small sites, read A/B testing with low traffic before accepting a 15% power test as normal.
- Report flat tests with their power. "No effect detected at 15% power" is a very different finding from "no effect detected at 80% power."
For the wider picture of planning, running and reading tests, start at the A/B testing hub.