A multi-armed bandit is a testing method that shifts traffic toward the better-performing variation while the test is still running, instead of holding a fixed 50/50 split until the end.
It trades some statistical certainty for fewer lost conversions, so use it when earning during the test matters more than learning why one version won, and use a classic A/B test when you need a reliable, measurable answer.
The name comes from slot machines ("one-armed bandits"): you face several machines with unknown payouts and must decide how much to keep testing new ones versus playing the one that looks best.
Statisticians call this the explore-exploit trade-off. Herbert Robbins framed it as a sequential design problem in a 1952 paper in the Bulletin of the American Mathematical Society.
How multi-armed bandit testing works
In a standard A/B test, each variation gets a fixed share of visitors for the whole run.
You only act on the data once the test reaches its planned sample size.
A bandit test starts the same way, with an even split.
Then it re-weights traffic at regular intervals based on results so far.
Variations that convert better get more visitors; weaker ones get fewer but are not switched off completely, so the algorithm can still notice if it guessed wrong early.
The three algorithms you will meet in testing tools are:
- Epsilon-greedy. A small fixed share of traffic (epsilon, for example 10%) is spread evenly across all variations to keep exploring. The rest goes to whichever variation currently has the best observed rate.
- Upper confidence bound (UCB1). Each round, pick the arm with the highest value of: observed mean + sqrt(2 × ln(total plays) / plays of that arm). Rarely tried arms get a bonus for uncertainty. Auer, Cesa-Bianchi and Fischer (2002) proved this simple rule keeps lost reward growing only logarithmically over time.
- Thompson sampling. Traffic is allocated in proportion to each variation's probability of being the best. This is the default in most modern tools, covered in the next section.
The quantity bandits try to minimize is called regret: the conversions you gave up by showing worse variations while you were still learning.
A fixed-split A/B test accepts high regret in exchange for an even, clean comparison.
What the vendors actually ship
Testing tools describe their bandits in their own docs, and the descriptions are candid about the limits:
- Optimizely uses Thompson sampling for binary metrics and epsilon-greedy for numeric metrics such as revenue. Its documentation says bandit results do not show statistical significance; they report "improvement over equal allocation" instead. You cannot change the primary metric once a bandit starts, and recent visitors are weighted more heavily to handle conversion rates that drift over time.
- VWO describes its bandit as an epsilon-greedy approach combined with Thompson sampling, and says it suits optimizers who "care more about maximizing a metric in a short time and can give up on statistical significance."
- Statsig Autotune (Statsig has been run by Amplitude since May 2026) uses Thompson sampling and eventually sends 100% of traffic to the winner. Its docs warn that Autotune cannot accurately measure a collection of metrics and is not a good way to understand secondary effects. More in our Statsig review.
Optimizely's own guidance places bandits in "optimization, not experimentation," best suited to short, temporary experiences such as headline tests or a holiday weekend sale.
Thompson sampling, explained with numbers
Thompson sampling is a Bayesian rule for choosing which variation to show next.
For each visitor, it draws one random plausible conversion rate for every variation from that variation's posterior distribution, then shows the variation whose draw is highest.
William R. Thompson proposed it in Biometrika in 1933, decades before web testing existed.
For conversion rates the math is short. Start each variation at Beta(1, 1), a flat prior.
After you observe conversions and non-conversions, the posterior is:
Beta(1 + conversions, 1 + non-conversions)
Because a variation's draw wins exactly as often as that variation is likely to be best, its traffic share ends up roughly equal to its probability of being best.
Uncertain arms still win draws sometimes, which is how exploration happens without a hand-set epsilon.
Worked example
Suppose after the first 2,000 visitors on a checkout test you have:
| Variation | Visitors | Conversions | Rate | Posterior |
|---|---|---|---|---|
| A (control) | 1,000 | 30 | 3.0% | Beta(31, 971) |
| B | 1,000 | 40 | 4.0% | Beta(41, 961) |
Drawing a large number of samples from both posteriors, B's draw beats A's about 89% of the time.
So Thompson sampling would now send roughly 89% of new visitors to B and 11% to A. A fixed-split test would keep sending 50% to A.
We ran a simple simulation to show the trade-off: 20,000 visitors, a true 3% rate for A and 4% for B, traffic re-weighted every 500 visitors, repeated 400 times.
| Approach | Average conversions | Share of traffic to B |
|---|---|---|
| Fixed 50/50 A/B test | 700 | 50% |
| Thompson sampling bandit | 777 | 89% on average |
| Perfect foresight (all traffic to B) | 800 | 100% |
The bandit recovered about three-quarters of the conversions the A/B test gave up.
The cost: A saw only around 2,200 visitors on average, so the estimate of A's rate, and therefore of the lift, is far noisier than in the 10,000-per-arm A/B test.
You gain conversions and lose a precise answer to "how much better is B?"
For a deeper treatment, the Tutorial on Thompson Sampling by Russo, Van Roy and colleagues covers when it works and when it fails.
Chapelle and Li (2011) tested it on display ad selection and news article recommendation and found it highly competitive with popular alternatives.
You can run the same Beta-posterior math on finished tests with our Bayesian A/B test calculator; the difference in philosophy is covered in Bayesian vs frequentist A/B testing.
Bandit testing vs A/B testing: when to use each
The choice comes down to one question: is the goal to earn as much as possible during the test, or to learn something you can trust and reuse?
| A/B test (fixed split) | Multi-armed bandit | |
|---|---|---|
| Main goal | Learn the true effect | Maximize one metric during the run |
| Traffic split | Fixed for the whole test | Shifts toward leaders |
| Statistical significance | Yes, with a planned sample size | Usually not reported |
| Lift estimate | Unbiased and precise | Noisy for losing arms |
| Secondary metrics | Reliable | Poorly measured |
| Cost of a bad variation | Full share of traffic for the whole run | Limited once it falls behind |
| Best for | Redesigns, pricing, checkout, anything you will keep | Short promotions, headlines, many similar options |
Use a bandit when the window is short
Liquid Death's in-house team added a countdown timer during a flash sale and saw a +9% conversion rate.
A sale that ends in days is the classic bandit case: there is no "later" in which to use a lesson, so every visitor sent to the weaker version during the sale is a loss you never recover.
The same logic applies to holiday banners, launch-week offers and news-style headlines.
If the experience will be gone before a fixed-horizon test would finish, a bandit is the better tool.
Check whether your traffic could finish one in time.
Use a bandit to cap losses across many weak options
Many variations lose. Mintminds tested four ways to move product specs on Lampenlicht's product pages, and all four produced significantly lower add-to-cart rates and revenue per visitor than the control.
MECLABS Institute ran four crowdsourced versions of a Consumer Reports donation email, and all four underperformed the control.
With a fixed split, each losing variant keeps its share of traffic until the test ends.
A bandit that includes the control would have drifted traffic back toward it as the losses showed up.
That is the real upside of bandits on A/B/n tests with many arms: fewer visitors spent on losers.
The catch is that you get a weaker read on why each one lost, which is the learning both of these teams were after.
Use an A/B test when you need the number and the side effects
Dave Diederen tested four homepage hero banner variations covering social proof position, trust badges, delivery and guarantee benefits, and CTA hierarchy.
The winner lifted conversion rate by +15.52%, revenue per visitor by +13.98% and profit per visitor by +14.29%.
A bandit could not have produced that report.
It optimizes one metric, so it would have chased conversion rate and left the revenue and profit numbers underpowered.
When a change is permanent, and you need to know it does not hurt AOV or margin, run a fixed split and watch your guardrail metrics.
The Optimizely-reported greeting card checkout test shows the flip side.
A bandit across several pricing and add-on variants produced +24% paid card purchases and +8.4% AOV, but the source only loosely describes which changes drove the result.
It is a good outcome and a weak lesson.
Where bandits go wrong
- Conversion rates change over time. If weekend visitors convert differently and one arm got most of its traffic on weekends, the pooled numbers can mislead. This is a form of Simpson's paradox, which Optimizely names in its docs as a reason it weights recent data more.
- Early luck locks in. A novelty effect can give a new variation a strong first week, and the bandit starves the control before the effect fades.
- Delayed conversions. Bandits update on what has converted so far. If purchases land days after the visit, as with subscriptions or B2B leads, the algorithm reacts to incomplete data.
- Very low traffic. Bandits do not create information. On a low-traffic site they still need enough conversions per arm to tell variations apart.
A practical pattern many teams use: run a fixed-split A/B test to find out whether an idea works, then use a bandit for short-lived, repeatable decisions such as which promo headline to show this week.
@DaveDiederen
OptimizelyBrowse more results with a clean control and variant in the full test database, or compare tools that offer bandits in our roundup of AI A/B testing tools.