A/B/n testing is an A/B test with more than one challenger: one control and two or more variants run at the same time, with traffic split between them, so you learn which version performs best in a single experiment.
A/B/C testing is the most common case, with exactly two variants against the control.
The cost is traffic: every extra arm needs its own sample, and comparing several variants to one control raises the chance of a false winner unless you correct for it.
What A/B/n testing is
The "n" stands for any number of versions.
In an A/B/n test, A is the control and B, C, D and so on are variants.
Each visitor is randomly assigned to one arm and stays there, exactly as in a standard A/B test.
What changes is the question. A plain A/B test asks "is B better than A?"
An A/B/n test asks "which of these options is best, and is it better than what we have now?"
That makes it the natural format when you have several credible answers to the same problem: three headlines, four price points, two layouts for the same module.
It is not the same as multivariate testing.
In an A/B/n test each variant is a whole version of the page, and the variants can differ from each other in any way.
A multivariate test changes several elements at once and builds every combination, so it can estimate how much each element contributes and whether the elements interact.
| A/B test | A/B/n test | Multivariate test | |
|---|---|---|---|
| Arms | Control + 1 variant | Control + 2 or more variants | Every combination of the elements tested |
| Question answered | Is this change better? | Which of these options is best? | Which element, and which combination, drives the result? |
| Traffic needed | Lowest | Grows with each arm | Highest; grows multiplicatively |
| Multiple-comparison risk | None from arms | Yes, one comparison per variant | Yes, many comparisons |
The line between the two can blur. Convertibles ran a homepage test for a luxury pajama brand as a 2x2 grid of content type and placement, which is a small multivariate design run as A/B/n.
The winner, press logos directly below the hero, was credited with +$109,829 per month, while the USP bar versions and lower placements lost.
A/B/C testing: the most common form
A/B/C testing means a control plus two variants.
Agencies use it a lot because it is the cheapest way to test two different answers to one problem without running two tests back to back.
Blend Commerce runs many of its Shopify tests this way, and the results show why the third arm is worth having.
In a cart drawer test for Fame, a dedicated Shop Pay button lifted conversion rate 47.75% and revenue per visitor 69.30%.
The other variant, payment icons, also raised conversion but cut checkout starts by 7%.
One A/B test of the icons alone would have told a muddier story.
The same thing happened on Singular Sound's product page.
A review about sound quality placed high on the page lifted conversion 19.14% in a 20-day A/B/C test.
A brand-trust review in the same spot raised add-to-cart 23.68% but lowered conversion 7.38%.
The two variants tested the same idea, "show a review up top," and got opposite results depending on which review it was.
ConvertiblesConvertibles
Blend CommerceBlend Commerce
GrowthRockThe statistical cost of extra variants
Each variant is compared with the control, so an A/B/n test with k variants runs k comparisons.
If each one uses a 5% significance level, the chance that at least one variant looks like a winner by luck alone rises with every arm. This is the multiple comparisons problem.
For independent comparisons, the family-wise error rate is:
FWER = 1 - (1 - alpha)^k
| Variants vs control (k) | Chance of at least one false winner at alpha = 0.05 | Bonferroni alpha per comparison |
|---|---|---|
| 1 (A/B) | 5.0% | 0.05 |
| 2 (A/B/C) | 9.8% | 0.025 |
| 3 (A/B/C/D) | 14.3% | 0.0167 |
| 4 (A/B/C/D/E) | 18.5% | 0.0125 |
| 5 | 22.6% | 0.01 |
The formula assumes independent comparisons. Comparisons that share a control are positively correlated, so the true rate is a little lower than the table, but it still climbs with each variant.
There are three common ways to handle it:
- Bonferroni: divide alpha by the number of comparisons. Simple and safe, but conservative, so it costs power.
- Dunnett's test: built for exactly this design, several treatments each compared with one control. Charles Dunnett published it in the Journal of the American Statistical Association in 1955. It uses the correlation from the shared control, so it is less conservative than Bonferroni.
- False discovery rate control: the Benjamini-Hochberg procedure (1995) limits the expected share of declared winners that are false, instead of the chance of any false winner. Optimizely's Stats Engine uses "a tiered version of the Benjamini-Hochberg procedure" to correct significance across metrics and variations, according to its documentation.
Check what your tool does before reading its results.
Some correct automatically; others report each variant against control at an uncorrected 95% and leave the adjustment to you.
Worked example: how much traffic an A/B/C test needs
Say your checkout converts at 3.0% and you want to detect a 10% relative lift (3.0% to 3.3%) with 80% power, two-sided. The standard two-proportion formula per arm is:
n = (z_(1-alpha/2) + z_(1-beta))^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2
| Design | Alpha per comparison | Visitors per arm | Total visitors | Days at 10,000 visitors/day |
|---|---|---|---|---|
| A/B | 0.05 | about 53,200 | about 106,400 | 11 |
| A/B/C, no correction | 0.05 | about 53,200 | about 159,600 | 16 |
| A/B/C, Bonferroni | 0.025 | about 64,400 | about 193,300 | 20 |
| A/B/C/D/E, Bonferroni | 0.0125 | about 75,600 | about 378,000 | 38 |
Going from two arms to five, with a correction, takes roughly 3.5 times the traffic of the original A/B test.
Run your own numbers in the sample size calculator, and see minimum detectable effect and statistical power for how those inputs trade off.
Sites with real volume can afford it. GrowthRock ran an A/B/C test for a high-traffic apparel retailer with 280,000 visitors per variation over two weeks: a "You Save #%" label in two colors against the original.
All three arms finished within 0.3% of each other.
With that sample, a flat result is a real answer, not a lack of data.
(Nearly every product had the same 30% discount at the time, which may explain why the label added nothing.)
Smaller samples need more caution. Wilson's Pet Food's homepage subcategory pills showed +22.86% revenue per visitor, but the A/B/C test was paused after 5,342 visitors at an 83.1% probability to beat control, and subscription revenue per visitor fell 15.73%.
Blend Commerce reports it as directional, which is the right reading.
When A/B/n testing is the right choice
- You have several distinct answers to one question. Convertibles tested a popup for an 8-figure dog treats brand with a control plus four variations. The three-free-treats offer won (+26% email capture, +$60,000 per month), and the full ranking came out as three gifts, then one gift, then $10 off $60, then 15% off. One test produced a ranked list, not just a yes or no.
- You are testing levels of a number. Prices, free-shipping thresholds and discount sizes suit A/B/n because the arms sit on a scale. Convertibles tested five price pairs for an adult wellness brand; moving two $99 products to $125 and $90 raised profit per visitor from $0.44 to $0.68 (+54.7%). See price testing for the setup.
- You want to protect against a single bad idea. Mintminds tested four ways to move product specs on Lampenlicht.nl's product pages, and all four significantly lowered add-to-cart and revenue per visitor. Testing several versions made the lesson clear: the original placement was doing its job.
When it is the wrong choice
- Low traffic. If an A/B test already takes four weeks, an A/B/C/D test with correction may take three months. Run fewer, bolder variants instead; see testing with low traffic.
- You want to know why something worked. If the variants differ in many ways at once, the winner tells you which page won, not which change mattered. Use a multivariate test or follow-up A/B tests.
- You care more about earning during the test than learning. A multi-armed bandit shifts traffic toward better arms as data comes in. It gives up clean inference in exchange for less traffic spent on losers.
- The variants are near-duplicates. Small tweaks of the same idea split your traffic without adding much to learn. The gin subscription test by Convertibles shows the pattern: a box contents preview lifted conversion 18% in both variants, and adding per-item retail values on top added no lift.
How to run an A/B/n test well
- Fix the number of arms and the split before launch. Changing allocation mid-test can bias results. Equal splits are the default and the easiest to analyze.
- Size the test for the corrected alpha. Calculate per-arm sample size with the alpha your tool will actually use, then multiply by the number of arms.
- Pick one primary metric. Variants times metrics multiplies comparisons. Optimizely's docs note that the more secondary metrics you add, the longer each may take to reach significance, though the primary metric is evaluated separately.
- Judge the winner against control, not just against the other variants. The best of four variants can still be worse than doing nothing, as the Lampenlicht.nl test shows.
- Expect the winning lift to shrink. Picking the best of several noisy estimates inflates it. The more arms, the bigger the winner's curse, so a confirmation test or holdout is worth considering for big decisions.
- Check the split. Uneven arm counts can signal a sample ratio mismatch, which breaks every comparison in the test.
Browse more multi-variant results, with who ran them and what they found, in the full test database.