A p-value is the probability of seeing a difference at least as large as the one you measured if your change actually did nothing.
In an A/B test, a p-value of 0.03 means that if the variant and control truly converted at the same rate, random noise alone would produce a gap this big or bigger about 3% of the time.
It is not the chance your variant is better, and it says nothing about how big the lift is.
This page explains how p-values work, using real tests from our database and their actual numbers.
For the wider topic of when to call a test, see statistical significance in A/B testing.
What is a p-value?
Every p-value starts with an assumption called the null hypothesis: the change has no effect.
The p-value asks how surprising your data would be if that assumption were true.
A small p-value means the data would be unusual under "no effect."
A large p-value means the data fit comfortably with "no effect." That is all it measures.
The American Statistical Association put this into a formal statement in 2016, with six principles. Three matter most for A/B testing:
- "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."
- "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."
- "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result."
Where the 0.05 cutoff comes from
The common 0.05 threshold is a convention, not a law.
Ronald Fisher wrote in Statistical Methods for Research Workers (1925) that the point where P = .05, "or 1 in 20," is "convenient to take" as a limit for judging significance.
Testing tools adopted it, and "95% significance" is just 1 minus 0.05.
You can pick a different threshold. What you cannot do is pick it after you see the data.
How the p-value works in an A/B test
For conversion rates, most tools use a two-proportion z-test (or a close relative). The steps fit in a small table.
| Step | Formula (plain text) |
|---|---|
| 1. Conversion rates | p1 = conversions A / visitors A; p2 = conversions B / visitors B |
| 2. Pooled rate | p = (conversions A + conversions B) / (visitors A + visitors B) |
| 3. Standard error | SE = sqrt( p × (1 - p) × (1/visitors A + 1/visitors B) ) |
| 4. Z-score | z = (p2 - p1) / SE |
| 5. Two-sided p-value | p-value = 2 × (1 - normal CDF of |z|) |
The z-score counts how many standard errors apart the two rates are.
Fisher's rule of thumb holds here: a z of about 1.96 gives a two-sided p-value of 0.05.
Worked example: a real reviews test
Convertibles ran a reviews test for an Australian whisky subscription brand, adding a "4.9 from 300+ reviews" row under the price and a review carousel below the buy button.
Over 15,615 visitors and 795 orders, conversion went from 4.725% to 5.460% (+15.56%).
The published data gives totals, not per-arm counts, so the calculation below is ours and assumes an even split of about 7,808 visitors per arm.
- Control: about 369 orders (4.725%). Variant: about 426 orders (5.460%).
- Pooled rate: 795 / 15,615 = 5.09%.
- SE = sqrt(0.0509 × 0.9491 × (1/7,808 + 1/7,808)) = about 0.00352, or 0.352 percentage points.
- z = (5.460% - 4.725%) / 0.352% = about 2.09.
- Two-sided p-value: about 0.037.
So if the reviews row did nothing, a gap this large would show up roughly 1 time in 27. That clears the 0.05 bar, though not by much.
Convertibles reported a 98% probability of being best, a Bayesian figure that lines up with the one-sided version of this p-value (about 0.018).
You can run the same numbers in our A/B test significance calculator.
"Probability to beat control" is not 1 minus the p-value
Many tools report a "probability to beat control" instead of a p-value.
Evan Miller notes in How Not To Run an A/B Test that dashboards often take the complement of the significance level and display it as a chance of beating the original. The two are easy to confuse.
Convertibles' sticky mobile CTA test for the same whisky brand shows why it matters.
Conversion rose from 1.135% to 1.366% (+20.4%) across 25,935 visitors and 327 orders, with a 95.1% probability to beat control.
Under our even-split assumption, that works out to z of about 1.67, a one-sided p-value of about 0.047 and a two-sided p-value of about 0.094. The same data is "95% significant" on one dashboard and fails a standard two-sided 0.05 test on another.
The test wasn't better or worse; the question being asked changed.
Decide before launch whether you are running a one-sided or two-sided test, and compare like with like.
ConvertiblesConvertibles
Clean Commit
Blend CommerceClean CommitBlend CommerceReading p-values in A/B test results
A p-value answers one narrow question. Most mistakes come from asking it to answer others.
A small p-value does not mean a big lift
The p-value mixes effect size and sample size.
A tiny lift on huge traffic can have a very small p-value, and a large lift on little traffic can have a large one.
To see how big the effect plausibly is, look at the confidence interval, not the p-value.
A large p-value does not mean "no effect"
A non-significant result usually means the test could not tell, not that the change did nothing.
Blend Commerce's mobile sticky add-to-cart test for PerTronix showed +10% conversion rate but was not statistically significant, and add-to-cart rate fell 20%.
The honest reading is "unknown," which is how our database counts it.
Compare that with Blend Commerce's value-props test for a premium travel equipment brand.
Across 99,754 visitors, conversion was 3.30% in both arms.
With that much traffic and an identical rate, "no meaningful difference" is a much safer conclusion than it would be on 5,000 visitors.
Low power makes borderline p-values unreliable
Clean Commit's condensed product gallery test reported +24.0% conversion rate and a 95.3% probability to beat control on 4,788 visitors.
The agency still labeled it a directional "positive signal," because statistical power was only 23%.
When power is that low, the results that happen to cross the threshold tend to overstate the true effect.
That is the winner's curse, and it is why a +24% lift on a small test should be expected to shrink on a retest.
Losses get p-values too
A two-sided test treats a drop the same as a gain.
Clean Commit's square product cards test lost: conversion fell 5.6% across 67,334 visitors, the original had a 98.2% probability of being better, and the result was significant.
A low p-value on a loss is just as useful, because it stops a bad change from shipping.
P-value mistakes in A/B testing
Peeking and stopping early
The p-value is only valid at the sample size you planned.
If you check results every day and stop the moment p drops below 0.05, false positives pile up.
In Evan Miller's worst-case example, checking after every observation pushed the real false positive rate to 26.1% when people expected 5%.
Miller's table shows how strict you would need to be to keep a true 5% rate: 2.9% reported significance if you peek once, 1.4% if you peek five times, and 1.0% if you peek ten times.
The fix is to set a sample size in advance, or to use a method built for monitoring.
Ramesh Johari, Leo Pekelis and David Walsh developed "always valid" p-values that stay correct under continuous monitoring, and the method was implemented in a large commercial testing platform. See sequential testing for how these work.
Testing many metrics or variants
Each comparison has its own 5% chance of a false alarm when nothing changed.
Check ten metrics and one is fairly likely to look "significant" by luck.
Pick one primary metric before launch, and if you must judge several, use a stricter threshold for each.
Treating 0.049 and 0.051 as opposites
A p-value of 0.049 and one of 0.051 are nearly identical evidence.
The ASA statement warns against basing decisions only on whether a threshold is crossed.
Weigh the interval, the cost of being wrong and whether the result makes sense.
When a p-value is the wrong tool
- Revenue per visitor and order value. These metrics are skewed by a few large orders, so a z-test on proportions does not apply. Use a t-test with care, or a bootstrap. See which statistical test to use.
- You want "how likely is B better?" A p-value cannot answer that. A Bayesian analysis can, given a prior.
- You must monitor continuously. Use a sequential method, not a fixed-sample p-value checked daily.
- Very low traffic. A p-value from an underpowered test is mostly noise. Test bigger changes or use a metric higher in the funnel.
- A broken split. If traffic did not split as planned, no p-value on the main metric can be trusted. Check for sample ratio mismatch first.
Used for what it measures, the p-value is a quick check on whether your result is distinguishable from noise.
Pair it with a planned sample size, a confidence interval and a single primary metric, and it does that job well.
For the full process, start at our A/B testing guide.