A Bayesian A/B test calculator takes the visitors and conversions for each version and returns the probability that the variant really converts better than the control, plus how much you stand to lose if you ship it and it turns out worse.
With 300 orders from 10,000 control visitors and 345 from 10,000 variant visitors, it says there is about a 96% chance the variant is better.
What a Bayesian A/B test calculator tells you
A significance calculator gives a p-value: how surprising your data would be if the versions were the same.
A Bayesian calculator answers what most people ask: how likely is it that B beats A, and by how much?
It reports three things:
- Probability to beat control: the share of plausible worlds, given your data, in which the variant's true conversion rate is higher.
- Expected loss: if you ship the variant and it is actually worse, how many conversion-rate points you lose on average, weighted by how likely that is.
- Credible interval for the lift: the range the true lift most likely falls in, for example a 95% chance it sits between two values.
Many testing tools report the first number under names like "probability to beat baseline" or "chance to win."
For the full trade-off, see Bayesian vs frequentist A/B testing.
The formula behind it
For a conversion rate (each visitor either converts or not), the standard model is a Beta distribution.
You start with a prior, Beta(a, b), then add what you observed:
- Posterior for A = Beta(a + conversions_A, b + visitors_A - conversions_A)
- Posterior for B = Beta(a + conversions_B, b + visitors_B - conversions_B)
With a flat prior, a = b = 1, which is the default in Evan Miller's derivation. It then computes:
| Output | Plain-text formula |
|---|---|
| Probability B beats A | P(p_B > p_A), where p_A and p_B are drawn from their posteriors |
| Expected loss of choosing B | E[max(p_A - p_B, 0)] |
| Expected loss of choosing A | E[max(p_B - p_A, 0)] |
| Relative lift | p_B / p_A - 1, summarized by its 2.5th, 50th and 97.5th percentiles |
Probability to beat has an exact closed form (Evan Miller's sum over Beta functions).
Expected loss and the interval are usually estimated by sampling each posterior many times.
Chris Stucchio's technical whitepaper for VWO sets out the expected-loss approach that many tools now use.
Inputs you need
- Visitors per version: unique users bucketed into each arm. If the split is far from what you set, run the SRM checker first.
- Conversions per version: users who completed the goal at least once. Count each user once.
- Prior (optional): leave it flat unless you have a good reason.
- Decision threshold: the probability you need before you act, often 95%, and the most loss you are willing to risk.
This model is for yes/no metrics such as conversion rate or add-to-cart rate.
Revenue per visitor needs a different model, because a few large orders swing it.
Worked example
Say a store tests a new product page.
Control gets 10,000 visitors and 300 orders (3.00%).
The variant gets 10,000 visitors and 345 orders (3.45%), an observed lift of 15%.
With flat Beta(1, 1) priors, our calculation gives:
| Output | Result |
|---|---|
| Posterior A | Beta(301, 9,701) |
| Posterior B | Beta(346, 9,656) |
| Probability B beats A | 96.4% |
| Expected loss if you ship B | 0.0035 percentage points |
| Expected loss if you keep A | 0.45 percentage points |
| 95% credible interval for lift | -1.2% to +33.9% (median +15.0%) |
| Probability the lift is above +5% | 87.9% |
For comparison, a two-proportion z-test on the same data gives z of about 1.80: a two-sided p-value of about 0.072 and a one-sided p of about 0.036. With a flat prior and this much data, the probability to beat comes out close to 1 minus the one-sided p-value.
The methods mostly differ in phrasing, not evidence.
VWO's published rule shows how expected loss gets used.
A variant counts as a "Smart Decision" when its probability to beat baseline is at least 95% and its expected loss is below a threshold of caring: current conversion rate x 0.0075 x 100 for non-revenue goals.
At a 3% baseline, that threshold is 0.0225 points. Our variant's 0.0035 is well under it.
A stronger prior, Beta(30, 970) on both arms (a 3% guess worth 1,000 visitors), pulls the estimate toward no difference: the probability drops to 95.8% and the median lift falls to +13.7%.
How to read the result
A 96% probability to beat does not mean a 96% chance of a 15% lift.
The credible interval above still runs from slightly negative to +34%.
Look at the probability, the interval and the expected loss together.
Real tests from our database show different readings:
- Convertibles' sticky mobile CTA on a whisky subscription homepage lifted conversion 20.4% with 25,935 visitors and 327 orders. That gave a 95.1% probability to beat control on conversion, but only 87.3% on revenue per visitor.
- Clean Commit's shorter mobile product gallery showed +24% conversion and a 95.3% probability to beat, from 4,788 visitors at just 23% power. Clean Commit labeled it a positive signal, not a proven win. A high probability on a small sample usually comes with a very wide interval.
- Convertibles' buy 3 get 1 free ball offer in the cart drawer sat at a 27% probability to beat control (-2.8% revenue per visitor) after 19 days and roughly 1,400-1,500 orders per version. It was stopped as not promising. A low reading tells you when to stop spending traffic.
Convertibles
Clean CommitConvertiblesClean CommitCommon mistakes
- Believing Bayesian tests are immune to peeking. They are not. In David Robinson's simulation of a change that actually hurt click-through, an expected-loss rule shipped the worse version in 2.5% of runs when checked once at day 20, and in 11.8% when checked daily with early stopping allowed. Set a minimum sample and run time first. The sample size calculator and how long to run a test help.
- Reading a high probability on tiny traffic as proof. Clean Commit's sitewide trust and design overhaul showed +66.4% conversions and a 99.1% probability to beat, with low power and a bundled redesign. Clean Commit itself counted it as unproven. Big early lifts tend to shrink (see the winner's curse).
- Feeding in sessions instead of users. The model treats each visitor as one independent yes/no trial.
When this calculator is the wrong choice
If you need a guaranteed false positive rate, for example for a regulated or high-stakes decision, use a fixed-horizon frequentist test or a proper sequential test.
The Bayesian probability is easier to explain, but it does not limit how often you ship a loser.
It is also wrong when traffic is too low to ever narrow the interval.
For more real results to judge your own idea against, browse the test database or start at the A/B testing guide.