The null hypothesis is the default claim that there is no real difference between the groups you are comparing.
In an A/B test it says the variant converts at the same rate as the control, so any gap you see is random noise.
You either reject it, because the data would be very unlikely if it were true, or you fail to reject it. You never prove it.
This page covers what the null and alternative hypotheses mean and how they work inside an A/B test, using real results from our database.
If you are looking for how to write a test idea ("if we change X, Y will improve because Z"), that is a different thing: see how to write an A/B test hypothesis.
Null hypothesis vs alternative hypothesis
Every significance test sets up two competing statements about the population, not about your sample.
| Null hypothesis (H0) | Alternative hypothesis (H1 or Ha) | |
|---|---|---|
| Plain meaning | Nothing is going on | Something is going on |
| A/B test form (two-sided) | p(variant) = p(control) | p(variant) ≠ p(control) |
| A/B test form (one-sided) | p(variant) ≤ p(control) | p(variant) > p(control) |
| What the test does with it | Assumes it is true and asks how surprising the data is | Accepted only indirectly, when H0 is rejected |
The idea of a null hypothesis comes from Ronald Fisher.
In The Design of Experiments (1935) Fisher wrote that the null hypothesis "is never proved or established, but is possibly disproved, in the course of experimentation." Fisher's tests had no formal alternative.
The alternative hypothesis came from Jerzy Neyman and Egon Pearson, whose 1933 paper framed testing as a choice between two hypotheses with two kinds of error.
That framework is what gives A/B testing its significance level (alpha) and its statistical power.
Modern A/B testing tools use a blend of both traditions.
Two practical points follow. First, the alternative must be chosen before the test starts: deciding to go one-sided after you see the variant ahead is a form of cheating.
Our page on one-tailed vs two-tailed tests covers when each is defensible.
Second, the alternative only says "different"; it says nothing about how big the difference is or whether it is worth shipping.
How the null hypothesis works in an A/B test
An A/B test runs the same logic every time:
- State H0: the variant and the control have the same true conversion rate.
- Pick alpha before launch, usually 0.05 (5%). This is the false-positive rate you accept if H0 is true.
- Run the test to the sample size you planned, then compute a test statistic and a p-value.
- If p < alpha, reject H0 and call the result statistically significant. Otherwise, fail to reject it.
The p-value is the probability of seeing a gap at least as large as yours if H0 were true.
Optimizely's documentation describes significance in exactly these terms: how unusual your results would be if there were no difference between variation and baseline.
It is not the probability that H0 is true.
The American Statistical Association's 2016 statement on p-values says this directly: p-values do not measure the probability that the hypothesis is true, or the probability that the data came from chance alone.
Worked example: a two-proportion z-test
Convertibles tested a sticky mobile CTA on a whisky subscription homepage.
They report 25,935 visitors, 327 orders, and conversion moving from 1.135% to 1.366% (+20.4%), with a 95.1% probability to beat control in their Bayesian tool.
Here is how the same data looks through a frequentist null-hypothesis test.
This is our calculation, assuming an even split of about 12,968 visitors per arm (roughly 147 and 177 orders):
| Step | Formula | Value |
|---|---|---|
| Observed difference | p2 − p1 | 0.01366 − 0.01135 = 0.00231 |
| Pooled rate (under H0) | p = (x1 + x2) / (n1 + n2) | about 0.0125 |
| Standard error | SE = sqrt(p(1 − p)(1/n1 + 1/n2)) | about 0.00138 |
| Test statistic | z = (p2 − p1) / SE | about 1.67 |
| Two-sided p-value | 2 × (1 − Φ(|z|)) | about 0.094 |
| One-sided p-value | 1 − Φ(z) | about 0.047 |
The pooled rate is used because the test assumes H0 is true: if both arms share one conversion rate, the best estimate of it is the combined rate.
At a two-sided alpha of 0.05, you fail to reject H0: 0.094 is above 0.05. With a one-sided test planned in advance, you would reject it.
The same data, read with a different alternative, gives a different verdict.
That is why the alternative has to be fixed before launch.
You can run your own numbers through our A/B test significance calculator.
When the z-test is the wrong choice
- Small counts. With only a handful of conversions per arm, the normal approximation breaks down. Use Fisher's exact test instead.
- Revenue per visitor or AOV. These are continuous and heavily skewed, not yes/no outcomes. A t-test (Welch's) or a bootstrap fits better.
- Checking results daily. A fixed-horizon test assumes you look once. If you stop as soon as p dips under 0.05, the real false-positive rate climbs well above 5%. Use a sequential test if you need to monitor.
- Many variants or metrics. Each extra comparison is another chance for a false rejection. Correct for multiple comparisons.
What "fail to reject the null" means in practice
Failing to reject H0 does not show the change did nothing.
It shows the test could not tell the difference from noise at the level of evidence you set. Our database has many results like this.
Blend Commerce added a block of value propositions below the add-to-cart button for a travel gear brand.
Across 99,754 visitors, conversion was flat at 3.30% in both arms, with about +2% in revenue per visitor that was not significant.
A large sample and an identical rate is about as close as A/B testing gets to "no meaningful effect", but it is still not proof of zero.
GrowthRock moved a "Free US Shipping & Returns" line under the add-to-cart button for a supplement brand.
After two weeks and more than 5,000 conversions, there was no difference in conversion rate, and other placements, including a promo bar, also showed none.
The opposite case is a big lift that still does not clear the bar.
Clean Commit moved a "Made in the USA" flag above the product title and saw +53.0% conversion rate on only 1,408 visitors.
The agency labeled it a positive signal, not a win.
On samples that small, a large swing is exactly what random noise produces, so H0 stays standing.
Blend Commerce's mobile sticky add-to-cart test for Pertronix showed +10% conversion and was extended, but still did not reach significance.
A 10% lift can be real and still go undetected when the test lacks power for an effect that size.
And GrowthRock's category link bar on an apparel mobile homepage showed +5% orders at 93% significance on a one-tailed basis, which did not reach significance under the tool's two-tailed test.
Like the worked example above, the verdict depended on which alternative was being tested.
Convertibles
Blend Commerce
Clean CommitBlend Commerce
GrowthRockGrowthRockErrors you can make about H0
| H0 is actually true | H0 is actually false | |
|---|---|---|
| You reject H0 | Type I error (false positive), rate = alpha | Correct: real effect found |
| You fail to reject H0 | Correct: no effect, none claimed | Type II error (false negative), rate = beta; power = 1 − beta |
Most "flat" tests in a typical program are a mix of true nulls and Type II errors from underpowered tests.
You cannot tell which from the p-value alone.
Planning a minimum detectable effect and sample size before launch is what keeps beta low.
More on both errors in Type I and Type II errors.
Common mistakes
- Saying "we accepted the null". Say "we failed to reject it". If you want to show two versions are equivalent, you need an equivalence test with a pre-set margin, not a non-significant p-value.
- Reading 1 − p as the chance the variant is better. A p-value of 0.03 does not mean a 97% chance of a real lift. Bayesian tools report that kind of probability directly, under their own assumptions, so treat that number as a different kind of answer.
- Treating rejection as proof the effect is large. Significant results from small tests tend to overstate the lift. Check the confidence interval, not just the verdict.
- Never checking your setup. In an A/A test H0 is true by design, which makes it a good check that your setup rejects H0 only about as often as alpha says it should.
Browse more real outcomes, including flat and losing tests, in the full test database, or start from our A/B testing guide.