An A/B test result is statistically significant when the gap between control and variant is too large to be random noise at the confidence level you chose, usually 95% (a p-value below 0.05). Enter visitors and conversions for each variant above, and the calculator runs a two-proportion z-test to give you the p-value and a significant or not significant verdict. A significant result only counts if you set your sample size before the test started.
Below we explain the method, take you through an example step by step, and show real results from our database of 285 ecommerce A/B tests where reading significance correctly mattered.
How the calculator works (visitors and conversions per variant)
You enter four numbers: visitors and conversions for control (A), and the same for the variant (B). The calculator compares the two conversion rates with a two-proportion z-test, which is the standard frequentist test for yes/no outcomes like "ordered" or "did not order."
The formula has four steps:
- Conversion rates: p(A) = conversions A / visitors A, and the same for B.
- Pooled rate: p = (conversions A + conversions B) / (visitors A + visitors B).
- Standard error: SE = √( p × (1 − p) × (1/visitors A + 1/visitors B) ).
- z-score: z = (p(B) − p(A)) / SE. The calculator turns z into a two-sided p-value. At 95% confidence, |z| needs to be above 1.96.
Worked example
Say each variant gets 10,000 visitors. Control has 300 orders (3.00%) and the variant has 345 (3.45%). That is a 15% relative lift, which looks like a clear win.
- Pooled rate: 645 / 20,000 = 3.225%
- Standard error: √(0.03225 × 0.96775 × 0.0002) = 0.0025
- z = 0.0045 / 0.0025 = 1.80, so p = 0.072
At 95% confidence, that is not significant. If the test had been planned for 20,000 visitors per variant and the same rates held (600 vs. 690 orders), z would be 2.55 and p would be 0.011. That would be significant. The lift is the same in both cases. All that changed is how much evidence you collected.
This is why you should size the test first. Use our A/B test sample size calculator and decide the smallest lift worth detecting before you launch.
What the p-value means
The p-value tells you how often you would see a gap at least this large if the change did nothing at all. A p-value of 0.072 means about 7% of the time. It is not "a 93% chance the variant is better."
The American Statistical Association makes the same point in its statement on p-values: a p-value does not measure the probability that a hypothesis is true, and it "does not measure the size of an effect or the importance of a result." Our guide to statistical significance covers this in more depth.
Many practitioners report confidence instead, which is 1 minus the p-value. Kasper Dolk reported an optimized site-wide navigation at +€127,112 revenue lift at 90% confidence, and swipeable collection-card images at +€96,350 at 97% confidence. Under a strict 95% rule, only the second one clears the bar. Whether 90% is good enough is a business call about how costly a wrong rollout would be.
Want results like this on your store?
Kasper Dolk at Maison Commerce. Kasper runs Maison Commerce, which builds CRO programs, landing pages and themes for fast-growing ecommerce brands such as ICON and HiStrips.
When to stop a test
Stop when you reach the sample size you planned, and not the first time the calculator turns green. If you check every day and stop at the first significant reading, your real false-positive rate climbs well past 5%. Evan Miller's "How Not To Run an A/B Test" shows one case where stopping at the first significant result pushed the false-positive rate to 26.1%, more than five times the intended level.
Our database has a clear example. Rubendgrf tested a doctor or expert endorsement badge. It showed +17% after week 1, but that was noise. After four weeks it settled at a flat +1.66% CVR. Stopping after week 1 would have shipped a "winner" that did very little.
The same idea can also win in one test and lose in another, so one significant result is not a rule. Dave Diederen saw +3.16% revenue per visitor from a checkout countdown timer, yet another checkout timer test lost 7.1% RPV, and a timer for a non-US European market lost 35.61% RPV.
Want results like this on your store?
Dave Diederen at Syntra Labs. Dave builds and tests Shopify stores at Syntra Labs, and shares most results in public.
Other practical rules: run for full weeks so weekday and weekend shoppers are both included, and don't change traffic splits during the test. How long to run an A/B test explains the timing. If you need to look early on purpose, use a tool built for it. Sequential testing and Bayesian methods are designed for that, and we compare them in Bayesian vs. frequentist A/B testing.
Revenue per visitor (RPV) significance
This calculator tests conversion rate. Revenue per visitor needs a different test, because it is an average of money amounts, not a yes/no outcome. Most visitors spend $0 and a few spend a lot, so the data is heavily skewed. A single large order can move the average.
The usual approach is to compare the two RPV means with a t-test. Welch's t-test is common because it does not assume both groups have equal variance. For very skewed revenue, analysts often use bootstrapping or cap extreme orders. To run any of these, you need order-level data, not just totals, so RPV significance is usually best left to your testing tool.
RPV and conversion rate can tell different stories. Elevate AB saw RPV rise from £3.32 to £3.60 (+£785 in 10 days) by moving the second product photo to the first position. Ten days is short for a revenue metric, so a result like this is worth re-checking over a longer window. Dave Diederen's pre-checked subscription option cut upfront RPV by 9.17% while subscription share rose 1,295%, which is a reminder to choose your primary metric before the test starts.
Both Convert and VWO include their own stats engines. Convert lists a frequentist and Bayesian stats engine plus sequential testing on its plans, starting at $399/month billed monthly or $299/month billed annually as of November 2026. VWO's pricing page describes a Bayesian-powered reporting dashboard with sequential and fixed-horizon tests, but does not list prices publicly. Neither is cheap for a small store. If you get under a few hundred orders a month, read A/B testing with low traffic before you pay for a tool. See our A/B testing tools roundup for more options.
Tools mentioned here
Questions
How do I determine if my A/B test is statistically significant?
Enter visitors and conversions for each variant into the calculator. It runs a two-proportion z-test and returns a p-value. If the p-value is below 0.05, the result is significant at 95% confidence, as long as you reached the sample size you planned before the test started.
Is this statistical significance calculator free?
Yes. The calculator is free to use with no sign-up. It works for conversion-rate tests with two variants.
How does sample size affect statistical significance?
The same lift can be significant or not depending on traffic. A 3.00% vs. 3.45% conversion rate gives p = 0.072 with 10,000 visitors per variant, but p = 0.011 with 20,000. Use a sample size calculator to plan traffic before launching.
Can I use this A/B test calculator for revenue per visitor?
Not directly. Revenue per visitor is a skewed average, so it needs a t-test such as Welch's, or bootstrapping, using order-level data. Most A/B testing tools calculate revenue significance for you.
Can I stop my A/B test as soon as it hits 95% confidence?
No. Checking repeatedly and stopping at the first significant result inflates false positives well above 5%. Run the test to its planned sample size, in full weeks, unless your tool uses sequential testing built for early looks.