Bayesian A/B testing tells you the probability that your variant beats the original, such as "92% chance B is better." Frequentist testing tells you how surprising your result would be if the change did nothing, and you call a winner when that p-value drops below a cutoff you set before the test. For most stores the method matters less than the discipline around it: enough orders, full weeks, and no stopping early because a number looked good.
How each method answers "did it win?"
Both methods look at the same data: visitors, orders and revenue in each group. They differ in the question they answer.
Frequentist: "Would this result be rare if nothing changed?"
A frequentist test starts by assuming the variant has no effect. It then asks how likely a gap this big would be by pure chance. That likelihood is the p-value. If it falls below 0.05, the result is called significant at 95% confidence.
The catch is that the 95% guarantee only holds if you pick your sample size in advance and look once at the end. Checking every morning and stopping on the first green day raises your false positive rate. Our guide to statistical significance covers this in more detail.
Bayesian: "How likely is it that B is better?"
A Bayesian test starts with a prior belief about conversion rates, then updates it with the data. The output is a probability statement that people actually want: "B has a 94% chance of beating A, and the lift is most likely between +2% and +11%."
That plain-English readout is why many store owners prefer it. But a Bayesian number is not a free pass to stop early either. A probability that swings from 60% to 95% and back in the first week is still noise.
What this looks like in real store tests
Practitioners in our database report results in both styles. Kasper Dolk reported a test of showing a testimonial vs no testimonial as "+7.22% profit per visitor, 94.4% probability to beat control." That is Bayesian phrasing. Under a strict 95% rule, it would still count as not quite there.
Kasper Dolk also reported a new theme with a BOGO offer at +44.54% conversion rate with a 99.7% probability to beat the old theme. Few people would argue with a number that high.
Other results are reported as confidence levels: an optimised site-wide navigation at +€127,112 revenue lift at 90% confidence, and expandable hero store benefits at +€49,938/month at 93.45% confidence. Both are below the classic 95% line, and a store would need to decide in advance whether 90% is good enough for them.
What the tools use
Most store owners never choose a method directly. They pick a tool, and the tool picks the stats. Here is what each vendor's own documentation says.
| Tool | Method | Main readout | Winner rule in the docs |
|---|---|---|---|
| VWO | Bayesian (SmartStats) | Probability to be the best | Chance to beat all of 95% or more, plus potential loss below a "threshold of caring" |
| Convert | Your choice: fixed-horizon frequentist, sequential, or Bayesian | P-values and confidence intervals, or probability to be best | 95% confidence by default for frequentist, customizable |
| Optimizely | Sequential frequentist (Stats Engine) | Always-valid significance | Results stay valid however often you check |
| AB Tasty | Bayesian | Chance to win | Green at 95% or higher |
| Intelligems | Bayesian | Probability to beat control | Read probabilities only after 300+ orders per group and 7+ days |
| Shoplift | Bayesian | Probability to Win | Must hold at a level for multiple days before a stage is assigned |
VWO. VWO's help center says SmartStats is Bayesian and runs 7 million simulated comparisons to estimate each variation's chance of being best. It also requires at least 25 conversions per variation, 1,500 visitors and a week of runtime before calling a winner. It does not mention a frequentist option, so it is not for teams who need p-values for a report. See our VWO vs Convert comparison.
Convert. Convert's support docs list three approaches: frequentist with p-values and multiple-testing correction, Bayesian with probability to be best, and sequential testing for early stopping. Convert's own blog describes the sequential option as using asymptotic confidence sequences. The downside of choice is that someone on your team has to make it, and switching methods mid-test to get a better-looking result defeats the purpose.
Optimizely. Optimizely's Stats Engine is frequentist but sequential. It is built on always-valid inference research from Stanford, so the significance number stays honest even if you check it daily. Optimizely is aimed at large companies, so it is usually not a fit for a small Shopify store.
AB Tasty. AB Tasty's docs describe a Bayesian "chance to win" that turns green at 95% or higher. A chance to win near 50% means the data shows no clear trend.
Shopify apps. Intelligems' docs describe a Bayesian model with Monte Carlo simulations and suggest waiting for 300+ orders per test group and 7+ days. Shoplift's docs say it uses Bayesian methods and that Probability to Win has to hold for multiple days before the app assigns a stage. Shoplift does not publish a single numeric cutoff, which some teams will find vague.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Also works well
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
For a wider view of features and fit, see our A/B testing tools roundup.
Practical differences for a store
On the same data with a sensible setup, Bayesian and frequentist tests usually reach the same conclusion. The differences show up in how people use them.
Readability
"94% chance to win" is easier to explain to a founder than "p = 0.06." That makes Bayesian output easier to act on, and also easier to over-trust. A 94% chance to win still means roughly 1 in 16 such calls points the wrong way.
Peeking and early stopping
Classic frequentist tests break if you peek and stop early. Sequential frequentist methods, like Optimizely's and Convert's sequential option, were built to fix that. Bayesian tools are often marketed as safe to peek, but their own docs still add guardrails: VWO's minimum week, Intelligems' 300 orders, Shoplift's multi-day hold.
Ruben saw why in a test of an authority badge with a doctor endorsement. It showed +17% after week 1, but that was noise, and the final result was +1.66% CVR, flat after 4 weeks. Any method would have given a wrong answer if the test had stopped on day 7.
Peter Christensen's customer trust count with 3 USPs on checkout moved checkout conversion from 69% to 76%, but it was still running when reported. A result in progress is a lead, not a verdict. Our guide on how long to run an A/B test covers stopping rules.
Results that don't repeat
Neither method tells you whether a win will hold on another store. Dave Diederen saw +3.16% revenue per visitor when adding a countdown timer to checkout, while a similar checkout countdown timer on another store lost 7.1% RPV. Ruben's countdown timer across several brands went +~10% CVR on a low-priced brand, flat on a high-priced brand and -9% CVR on another.
A 95% probability or a 95% confidence result describes one test on one audience. It is not a law. This is why we publish full sets of wins, losses and flat results in our test database.
Low traffic
Bayesian tools will give you a probability on any amount of data, which feels helpful on a small store. But a 70% chance to win from 40 orders tells you very little. With low traffic, test bigger changes so the effect is large enough to detect, and check your numbers in a significance calculator before you ship.
Which to choose
For most stores, the honest answer is: use whatever your tool uses, and set your rules before the test starts. A good process with either method beats a perfect method used badly.
- Choose Bayesian if you want results the whole team can read, you make business calls where 90% sure is good enough, and your tool (VWO, AB Tasty, Intelligems, Shoplift) already uses it. Still commit to a minimum run time and order count.
- Choose fixed-horizon frequentist if you can calculate a sample size in advance and will wait until you reach it. It is the most widely understood method and the easiest to audit.
- Choose sequential frequentist if your team checks results daily and wants to stop early when the evidence is strong, without inflating false positives. Convert and Optimizely offer it.
Whatever you pick, write down three things before launch: the main metric (we suggest revenue per visitor for most stores), the minimum run time in full weeks, and the threshold you will act on. Then don't change them mid-test.
If you want help setting up a testing program with clear stopping rules, Kasper Dolk, whose probability-based results are cited above, builds CRO programs for ecommerce brands.
Want results like this on your store?
Kasper Dolk at Maison Commerce. Kasper runs Maison Commerce, which builds CRO programs, landing pages and themes for fast-growing ecommerce brands such as ICON and HiStrips.
Questions
What is Bayesian A/B testing?
Bayesian A/B testing combines a prior belief about conversion rates with your test data to give the probability that a variant beats the original, such as a 94% chance to win. Tools like VWO, AB Tasty, Intelligems and Shoplift use it.
What is the difference between Bayesian and frequentist A/B testing?
Frequentist testing asks how surprising your result would be if the change did nothing and reports a p-value. Bayesian testing reports the probability that one version is better. On the same data with a sound setup, they usually agree.
Can I stop a Bayesian A/B test early?
You can check it at any time, but stopping on an early spike is still risky. Vendor docs add guardrails, such as VWO's minimum of a week and 25 conversions per variation, and Intelligems' suggestion of 300+ orders per group and 7+ days.
What probability is enough to call a Bayesian A/B test?
VWO and AB Tasty both use 95% as their default winner line. Some stores act at 90% for low-risk changes. Pick your threshold before the test starts and stick to it.
Is Bayesian better than frequentist for low-traffic stores?
Not really. A Bayesian tool will show a probability on little data, but it will be uncertain. Low-traffic stores do better by testing bigger changes and running tests for full weeks.
Sources
- help.wingify.com/hc/en-us/articles/59066062029977-How-Wingify-Calculates-a-Winning-Variati
- support.convert.com/hc/en-us/articles/statistical-models-in-convertcoms-ab-testing-platfor
- www.convert.com/blog/a-b-testing/frequentist-vs-bayesian-ab-testing/
- arxiv.org/pdf/1512.04922
- certification.optimizely.com/concepts/experimentation/stats-engine-deep-dive/
- docs.abtasty.com/reporting-and-performances/reporting/understand-the-statistics-behind-you
- docs.intelligems.io/analytics/statistical-significance
- docs.shoplift.ai/reporting/bayesian-methodology