The Bonferroni correction is the simplest fix for the multiple comparisons problem.
You divide your significance threshold by the number of comparisons you make: with 5 comparisons at alpha = 0.05, each one needs p < 0.01 to count as a win.
That keeps the chance of even one false positive across the whole set at or below 5%.
The cost is lost power, which is why many A/B testing tools use Holm or false discovery rate (FDR) methods instead.
This page covers why extra variants, metrics and segments inflate false positives, how Bonferroni, Holm and Benjamini-Hochberg compare, and which one fits a given A/B test.
Along the way we point to real tests from our database where several comparisons came out of one experiment.
The multiple comparisons problem: why more variants and metrics mean more false wins
A single comparison at alpha = 0.05 has a 5% chance of a false positive when the change truly does nothing.
That is the Type I error rate you sign up for when you call a result statistically significant.
Run several comparisons and those 5% chances add up.
If the comparisons are independent and none of the changes has any real effect, the chance of at least one false win is:
P(at least one false positive) = 1 - (1 - alpha)^m, where m is the number of comparisons.
| Comparisons (m) | Example setup | Chance of at least one false win at alpha = 0.05 |
|---|---|---|
| 1 | A/B test, one metric | 5.0% |
| 4 | 4 variants vs control, one metric | 18.5% |
| 12 | 4 variants x 3 metrics | 46.0% |
| 20 | 4 variants x 5 metrics | 64.2% |
Real comparisons are rarely fully independent (conversion rate and revenue per visitor move together), so the true figure is usually a bit lower.
The direction doesn't change: every variant, metric and segment you add is another chance to find a "winner" that is only noise.
In CRO the comparisons pile up in four places:
- Variants. An A/B/n test with four challengers is four comparisons against control.
- Metrics. Conversion rate, revenue per visitor, AOV, add-to-cart rate and clicks are each a separate test.
- Segments. Mobile vs desktop, new vs returning, traffic source. Each slice multiplies the count.
- Peeking. Checking a fixed-horizon test every day and stopping when it crosses p < 0.05 is repeated testing over time. Sequential testing is built for that case.
What this looks like in real tests
Plenty of tests in our database involve several comparisons at once.
Dave Diederen tested 4 homepage hero banner variations for a Shopify brand and reported +15.52% conversion rate, +13.98% revenue per visitor and +14.29% profit per visitor for the best version.
That is four challengers and three metrics from one experiment.
The source doesn't say whether a correction was applied.
Another Dave Diederen test increased font size with +2px and +4px variants for an older audience: control vs +2px vs +4px, reported as +2.12% conversion rate, +3.82% revenue per visitor and +1.66% AOV.
AEZAKMI Group tested six onboarding flows in its LED controller app, moving conversion to subscription from 0.8% to 1.29% (+61%) and ARPU from $0.12 to $0.14.
The more options you test, the bigger the chance that the top one is partly luck.
That is the winner's curse, and it is why the best of six usually shrinks a little after launch.
Many comparisons can also all point the same way, which is evidence in its own right.
Mintminds ran four ways to move product specs on Lampenlicht.nl, and all four had significantly lower add-to-cart and revenue per visitor.
One loss might be noise. Four losses in a row tell you the original layout was doing its job.
False discovery rate vs familywise error rate
Multiple testing corrections control one of two error rates, and picking a method mostly means picking which one you care about.
| Error rate | What it controls | Typical method | Plain-English promise |
|---|---|---|---|
| Familywise error rate (FWER) | The chance of one or more false positives across all comparisons | Bonferroni, Holm | "There's at most a 5% chance that any of my wins is fake." |
| False discovery rate (FDR) | The expected share of your declared wins that are false | Benjamini-Hochberg | "On average, no more than 5% of the wins I declare are fake." |
The false discovery rate was introduced by Yoav Benjamini and Yosef Hochberg in a 1995 paper in the Journal of the Royal Statistical Society, Series B.
They showed that FDR equals FWER when every null hypothesis is true, and is smaller otherwise, so controlling it leaves more power to find the real effects.
FDR is the more forgiving standard. If you declare 20 wins under 5% FDR, you expect about one of them to be false.
Under FWER control you'd expect less than a 5% chance that even one is false, but you'd also declare fewer wins.
Multiple testing correction methods: Bonferroni, Holm and Benjamini-Hochberg
The Bonferroni correction is named after the Italian mathematician Carlo Emilio Bonferroni, whose probability inequality it rests on.
Olive Jean Dunn turned it into a practical method for comparing means in a 1961 Journal of the American Statistical Association paper, which is why it is sometimes called the Dunn-Bonferroni method (CDC's Emerging Infectious Diseases etymology note).
Sture Holm published the step-down version in the Scandinavian Journal of Statistics in 1979 as a "sequentially rejective" Bonferroni test.
It controls FWER just like Bonferroni but never rejects fewer hypotheses.
How each method works
| Method | Controls | Rule (m comparisons, target alpha or q) |
|---|---|---|
| Bonferroni | FWER | Call a comparison significant if p < alpha / m. |
| Holm (Holm-Bonferroni) | FWER | Sort p-values from smallest to largest. Compare the k-th smallest to alpha / (m - k + 1). Keep rejecting until the first one fails, then stop. |
| Benjamini-Hochberg | FDR | Sort p-values from smallest to largest. Find the largest k where p(k) ≤ (k / m) x q. Call comparisons 1 to k significant. |
Worked example: five comparisons, three answers
Say you run a test with control plus five variants, one primary metric, and get these five p-values (sorted): 0.003, 0.012, 0.021, 0.040, 0.300. Target alpha (or q) is 0.05. Without any correction, four of the five look like wins.
| Rank k | p-value | Bonferroni threshold (0.05/5) | Holm threshold (0.05/(5-k+1)) | BH threshold ((k/5) x 0.05) |
|---|---|---|---|---|
| 1 | 0.003 | 0.010: significant | 0.0100: significant | 0.010: significant |
| 2 | 0.012 | 0.010: not | 0.0125: significant | 0.020: significant |
| 3 | 0.021 | 0.010: not | 0.0167: not (stop here) | 0.030: significant |
| 4 | 0.040 | 0.010: not | not tested | 0.040: significant |
| 5 | 0.300 | 0.010: not | not tested | 0.050: not |
Result: Bonferroni keeps 1 winner, Holm keeps 2, and Benjamini-Hochberg keeps 4. Same data, three answers.
That is why you should choose the method before the test starts, not after you see which one gives you the result you want.
The price of correcting: power
A stricter threshold means you need more traffic to detect the same lift.
Moving from alpha = 0.05 to 0.01 (Bonferroni with five comparisons) raises the sample size needed for 80% power by roughly half for a two-sided test.
Plan for it up front with the sample size calculator by entering the corrected alpha, not 0.05.
Which correction to use in an A/B test
There's no single right answer, but there are sensible defaults.
- One primary metric, one variant: no correction is needed for the decision. Treat everything else as supporting evidence.
- A/B/n test where you'll ship one winner: correct across variants. Holm beats plain Bonferroni because it has more power and the same guarantee.
- Many secondary metrics or an exploratory readout: Benjamini-Hochberg. You accept a few false leads in exchange for finding more real ones, then confirm the important ones in a follow-up test.
- A false positive would be costly (a price change, a checkout rebuild, a legal claim): FWER control with Bonferroni or Holm.
When each method is the wrong choice
- Bonferroni is wrong when you have many correlated metrics (CR, RPV and orders all tracking the same purchases). It assumes the worst case and throws away power. Holm is never worse, so there's little reason to use plain Bonferroni except for its simplicity.
- Benjamini-Hochberg is wrong when a single false win is expensive, and when the tests are negatively correlated. The original proof covers independent tests, and GrowthBook's docs note it assumes tests are "independent or positively correlated."
- Any correction is wrong for guardrail metrics if it makes harm harder to detect. With guardrails you want to catch damage, so a looser threshold is often the safer call.
- Correcting is not a fix for peeking. Bonferroni over a few planned looks is very conservative; use a sequential method instead.
What testing tools do by default
Tools handle this differently, so check what your report is doing before you read it.
| Tool | What the docs say |
|---|---|
| Optimizely | Uses "a tiered version of the Benjamini-Hochberg procedure" across metrics and variations, treats the primary metric separately from secondary and monitoring metrics, and does not maintain FDR control when you segment results (Optimizely docs). |
| GrowthBook | Lets you pick no correction, Holm-Bonferroni or Benjamini-Hochberg, for the frequentist engine only. The correction covers goal metrics and variations, not secondary or guardrail metrics (GrowthBook docs). |
| Statsig | Offers Bonferroni (applied across test groups and/or scorecard metrics) or Benjamini-Hochberg, but not both on one experiment (Statsig docs). |
Segments and secondary metrics: the hidden comparisons
Segment results are where uncorrected comparisons cause the most trouble, because people slice after the fact.
Optimizely's docs say plainly that FDR control is not maintained once you segment.
Some device splits in our database are big enough to act on.
Blend Commerce's two-image product cards for Stone Creek Coffee went +20% conversion rate on mobile and -12% on desktop, and the team shipped it on mobile only.
Blend Commerce's simplified apparel homepage hero lifted desktop conversion 26% while mobile declined, and it was also shipped on desktop only.
Those calls may be right. But a segment result you found by looking through every slice should be treated as a new hypothesis, not a finding.
Rerun it as a test targeted at that segment, or at least confirm it holds up with a corrected threshold. More on this in segmentation.
Secondary metrics cause the opposite problem: a significant lift on a minor metric can hide a loss on the one that pays.
Clean Commit's wider collection page produced a statistically significant +4.0% click-through rate, while conversion fell 5.9%, orders 8.0% and revenue per visitor 6.1%.
Their sticky add-to-cart bar test lifted desktop add-to-cart 45.3% while conversion fell.
The fix is to name one primary metric before launch and decide on it.
Pre-register which metrics and segments you'll look at, and how many comparisons that makes.
Then the correction is a formality rather than a debate.
@DaveDiederen@DaveDiederen
Blend Commerce
Clean Commit