Statistical significance in an A/B test means the gap between your control and your variant is big enough, given how many visitors you had, that random chance alone would rarely produce it. Most tools call a result significant at 95% confidence, which means a p-value below 0.05. You can trust a significant result when you fixed the sample size in advance, ran the test to that size without stopping early, and the lift is large enough to matter to your business.
The rest of this guide explains each part in plain words, with real results from our database of 285 ecommerce A/B tests showing where significance helps and where it misleads.
What a p-value means, in plain words
Every A/B test starts from a boring assumption: the change does nothing, and any difference you see is noise. Statisticians call this the null hypothesis.
The p-value answers one question. If the change really did nothing, how often would you see a gap at least this large just from the random mix of visitors who landed in each group? A p-value of 0.03 means about 3% of the time.
A low p-value says your data would be surprising if nothing had changed. It does not say "there is a 97% chance the variant is better." That is the most common misreading, and the American Statistical Association warned against it in its 2016 statement on p-values: p-values do not measure the probability that a hypothesis is true, and they do not measure the size of an effect.
A worked example
Say your product page converts at 3.0%. You test a new headline and split traffic evenly.
- With 10,000 visitors per side, control gets 300 orders (3.0%) and the variant gets 345 (3.45%). That is a 15% relative lift, but a standard two-proportion z-test gives a p-value of about 0.07. Not significant at 95%.
- With 20,000 visitors per side and the same rates, control gets 600 orders and the variant gets 690. The p-value drops to about 0.01. Significant.
The lift did not change. Only the amount of evidence did. That is why a big percentage on a small sample means little, and why you should work out your sample size before you launch. You can check your own numbers in the calculator linked at the end of this guide.
Confidence level and power
Two settings decide how strict your test is. They protect you from two different mistakes.
| Setting | Common default | What it protects against | The mistake in plain words |
|---|---|---|---|
| Confidence level (1 minus alpha) | 95% | False positive (Type I error) | You ship a "winner" that does nothing |
| Statistical power (1 minus beta) | 80% | False negative (Type II error) | You miss a change that really works |
The 95% and 80% figures are habits, not laws. As the Stack Overflow data team put it, 80% power and a 5% significance level are typical standards that carry "a fair amount of baggage and tradition." Convert's documentation also uses 95% as its default confidence level, and you can change it.
Why power gets ignored
Store owners watch confidence closely and forget power. But an underpowered test usually ends "flat," and flat does not mean "no effect." It often means "not enough visitors to tell."
In the example above, detecting a 15% lift on a 3% conversion rate at 95% confidence and 80% power takes roughly 24,000 visitors per variant. The smaller the lift you want to detect, the more traffic you need. Our guide to the minimum detectable effect walks through this.
What confidence levels look like in practice
Some practitioners in our database report the confidence level next to each result, which makes the trade-off visible. Kasper Dolk's tests at Maison Commerce show a range:
- A second-threshold mystery gift in the cart returned +€30,578 revenue lift at 99% confidence.
- Making hero store benefits expandable returned +€49,938/month at 93.45% confidence.
- An optimized site-wide navigation returned +€127,112 revenue lift at 90% confidence.
- A product finder quiz on the collection page showed +€54,910 revenue lift at 89% confidence, marked as still in progress.
Only the first clears the usual 95% bar. That does not make the others worthless. A team can choose to ship at 90% when the change is cheap to reverse and the downside is small. What matters is picking the bar before the test starts and saying it out loud, as these results do.
Want results like this on your store?
Kasper Dolk at Maison Commerce. Kasper runs Maison Commerce, which builds CRO programs, landing pages and themes for fast-growing ecommerce brands such as ICON and HiStrips.
Why peeking breaks results
Peeking means checking your results while the test runs and stopping as soon as the tool says "significant." It feels efficient. It quietly wrecks the math.
A standard significance test assumes you fixed the sample size in advance and looked once at the end. Each extra look is another chance for random noise to cross the line. Evan Miller's well-known essay How Not To Run an A/B Test shows that if you stop the moment a test hits 5% significance, you can wrongly find a significant result about 26.1% of the time in the example it simulates, more than five times the rate you thought you were accepting.
Our database has a clean example. Ruben, who runs CRO at Tailored Commerce, added a doctor and expert endorsement badge to a product page. After week one it looked like +17%. After four weeks it settled at +1.66% CVR, effectively flat. Ruben's own note on the result: the week-one number was noise. Anyone who stopped after seven days would have shipped a "17% winner" that did almost nothing.
Want results like this on your store?
Ruben at Tailored Commerce. Ruben runs CRO at Tailored Commerce for brands like Hears, Moov and FLÂNEUR.
How to avoid it
- Calculate the sample size before launch, then do not judge the result until you reach it.
- Run in full weeks, so weekday and weekend shoppers are both counted. Our guide on how long to run an A/B test covers this in detail.
- Look at the dashboard for bugs and broken tracking, not for a verdict.
- If you need to monitor results as they come in, use a method built for it. Convert, for example, offers a sequential testing mode that its docs describe as allowing "early stopping" while keeping statistical validity, alongside fixed-horizon frequentist and Bayesian options. Our Bayesian vs frequentist guide explains the differences.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
Significance vs business impact
A significant result tells you the effect is probably not zero. It does not tell you the effect is big, profitable, or the same next month. The ASA statement says it directly: statistical significance "does not measure the size of an effect or the importance of a result."
Pick the right metric first
A test can be significant on one metric and bad for the business on another. When Dave Diederen pre-checked the subscription option by default, revenue per visitor fell 9.17% upfront while subscription share rose 1,295%. Whether that is a win depends on lifetime value, not on the p-value. Choose your primary metric before launch. Our A/B testing metrics guide covers conversion rate, revenue per visitor and guardrail metrics.
A result applies to the store where it ran
Significance is about one test, on one store, in one period. Ruben tested a countdown timer across several brands and got +~10% CVR on a low-priced brand, flat on a high-priced brand, and -9% CVR on another. Same idea, three different answers. A significant win elsewhere is a good hypothesis for your store, not a guarantee.
Flat and losing results are information too
Plenty of well-run tests end without a winner. Dave Diederen's 'Clinician Recommended' badge came back flat, and adding small customer profile icons next to the star rating produced -3.84% RPV. Both told the team where not to spend more time.
A quick checklist before you trust a result
- Was the sample size set before launch, and was it reached?
- Did the test run for full weeks?
- Did it clear the confidence level you chose in advance?
- Is the lift big enough to be worth the cost of building and keeping the change?
- Did guardrail metrics, such as average order value, hold up?
- Was traffic split as expected, with no tracking bugs?
If any answer is no, treat the result as a lead to retest, not a fact. Our list of common A/B testing mistakes covers more of these traps.
Check your own test with the calculator
To check whether a finished test is significant, enter visitors and conversions for each variant into our A/B test significance calculator. It returns the p-value, the confidence level and the relative lift.
Before your next test, use the sample size calculator to see how many visitors you need for the lift you hope to detect. If the answer is more traffic than you get in a month or two, test bolder changes or pick a higher-traffic page. Then browse the full test database for ideas that have worked, and failed, on other stores.
Questions
What is statistical significance in A/B testing?
It means the difference between your control and variant is large enough, given your sample size, that random chance alone would rarely produce it. Most tools call a result significant when the p-value is below 0.05, which equals 95% confidence.
What confidence level should I use for an A/B test?
95% is the common default, and many tools, including Convert, use it out of the box. Some teams accept 90% for cheap, easy-to-reverse changes. Pick the level before the test starts, not after you see the results.
Does a 95% significant result mean there is a 95% chance the variant is better?
No. A p-value of 0.05 means that if the change did nothing, you would see a gap this large about 5% of the time. It does not give the probability that the variant is better, and it does not measure how big the effect is.
Can I stop an A/B test as soon as it reaches significance?
Not with a standard fixed-sample test. Stopping the moment a result looks significant inflates false positives a lot. Set the sample size in advance and wait for it, or use a sequential method built for monitoring results as they come in.
Why is my A/B test not reaching statistical significance?
Usually the sample is too small for the size of the effect, or the change truly has little effect. Use a sample size calculator before launch, run in full weeks, and test bolder changes if your traffic is low.