To analyze A/B test results, first confirm the data is trustworthy (the traffic split matches what you set and the test ran its planned length), then compare the primary metric with a significance test and a confidence interval, check guardrail metrics, and only then look at segments.
A result is a win only if the primary metric clears the significance bar you set before launch; anything else is a loss or an inconclusive test, and both still tell you something.
This guide walks through each step, shows a worked example with numbers, and uses real results from our database of A/B tests to show where analysis usually goes wrong.
What goes in an A/B test report
An A/B test report is the record of one test that someone outside the team can read and trust.
Write it so a person who joins next year can tell what you changed, what happened, and why you shipped or didn't.
A useful report has these parts:
- Hypothesis: the change, the metric you expect it to move, and why.
- Setup: page, audience, devices, traffic split, start and end dates, and the sample size you planned.
- Primary metric: the one metric that decides the test, named before launch.
- Guardrail metrics: the metrics that must not get worse, such as revenue per visitor or refund rate.
- Results: visitors and conversions per arm, the relative lift, the p-value or probability, and the confidence interval.
- Health checks: the sample ratio check and any tracking or QA problems.
- Decision and next step: ship, don't ship, or retest, and what you learned either way.
The raw counts matter most. A line like "+53% conversion" means little without the visitors behind it, as one of the examples below shows.
How to analyze A/B test results, step by step
Do these steps in order. Looking at the lift first makes it tempting to explain away problems in the later checks.
- Check the test ran as planned. Did it reach the sample size and duration you set? Stopping early because the dashboard looked good inflates false positives. Evan Miller showed that checking for significance after every observation can push a nominal 5% false positive rate to 26.1% in a worst case.
- Check the traffic split. If you set 50/50 and got something else, you may have a sample ratio mismatch. Run a chi-square goodness-of-fit test on visitors per arm; a p-value below 0.001 is the usual alarm. This is not rare: a KDD 2019 paper reported SRMs in about 6% of experiments at Microsoft. When there's an SRM, don't trust the lift until you find the cause.
- Compare the primary metric. Calculate the conversion rate per arm, the relative lift, the p-value, and a confidence interval. Our significance calculator does this from raw counts.
- Read the confidence interval, not just the p-value. The interval shows the range of effects the data supports. A "significant" result whose interval runs from +1% to +25% means you know the direction, not the size. See confidence intervals and statistical significance.
- Check guardrails. A win on the primary metric can hide a loss somewhere that matters more. Look at revenue per visitor, average order value, and downstream steps. More on this in guardrail metrics.
- Look at segments last, as a source of new hypotheses, not new winners (see below).
- Decide and write it down. Record the decision in the report, including the tests you didn't ship.
Anatta's buy-box test is a good model of steps 1 to 3. The report gives the real counts (34,730 control and 6,144 variant visitors on an intended 85/15 split), notes that the SRM check passed, and reports AOV moving from $102.63 to $104.74 with a 97.6% Bayesian probability of a positive effect. See the tabbed buy box test.
A worked A/B test analysis with numbers
Here is a hypothetical checkout test analyzed end to end.
You planned a 50/50 split and 20,000 visitors per arm.
| Visitors | Orders | Conversion rate | |
|---|---|---|---|
| Control (A) | 20,000 | 600 | 3.0% |
| Variant (B) | 20,210 | 687 | 3.4% |
SRM check. Expected visitors per arm = 40,210 / 2 = 20,105. Chi-square = (20,000 - 20,105)^2 / 20,105 + (20,210 - 20,105)^2 / 20,105 = 1.10. With one degree of freedom, p = 0.29. That is far above 0.001, so the split is fine.
Lift. (3.4% - 3.0%) / 3.0% = +13.3% relative, or +0.4 percentage points.
Significance (two-proportion z-test). Pooled rate p = (600 + 687) / 40,210 = 3.2%.
Standard error = sqrt(p x (1 - p) x (1/n_A + 1/n_B)) = sqrt(0.032 x 0.968 x (1/20,000 + 1/20,210)) = 0.00176. z = (0.0340 - 0.0300) / 0.00176 = 2.27. The two-tailed p-value is 0.023, below the 0.05 threshold.
95% confidence interval. Unpooled standard error = sqrt(0.03 x 0.97 / 20,000 + 0.034 x 0.966 / 20,210) = 0.00175. The difference is 0.0040 ± 1.96 x 0.00175, or +0.06 to +0.74 percentage points.
In relative terms that's roughly +1.8% to +24.8%.
Reading it. The variant very likely helps, but the true lift could be anywhere from tiny to large.
Forecast with the low end, not the +13.3% point estimate.
Significant results from tests near the power limit tend to overstate the effect, which is the winner's curse.
When this method is the wrong choice. The z-test fits conversion rates with large samples.
It's wrong for revenue or AOV, which are skewed by big orders (use a t-test on per-visitor values, a bootstrap, or trim outliers as Anatta did).
It's wrong if you checked results repeatedly and stopped at the first significant reading (use a sequential method instead).
And with very small counts per arm, use Fisher's exact test, since the normal approximation breaks down.
Post-test segmentation: useful, but easy to abuse
Post-test segmentation means splitting results after the test by device, new vs. returning visitors, traffic source, or country.
It can reveal real differences. It can also manufacture fake ones.
The problem is multiple comparisons. Check 10 segments at a 0.05 threshold on a change that does nothing, and you have about a 40% chance (1 - 0.95^10) that at least one looks significant.
A simple fix is the Bonferroni correction: divide 0.05 by the number of segments you check.
With 10 segments, each needs p below 0.005. Each segment also has far fewer visitors than the full test, so it has less power and wider intervals.
Device splits are the one place segmentation often pays off, because mobile and desktop really are different pages.
Blend Commerce saw this on Stone Creek Coffee's product cards: showing multiple images lifted mobile conversion 20% and RPV 11%, while desktop conversion fell 12% and RPV 7%.
They shipped on mobile only. See the multi-image product card test.
Rules that keep segment analysis honest:
- Name the segments you care about before launch, with a reason.
- Treat an unplanned segment result as a new hypothesis, then retest it on that segment alone.
- Check that the segment has enough visitors to say anything.
- Watch for Simpson's paradox, where a mix shift between segments makes the total point the other way.
What to do with an inconclusive A/B test
An inconclusive A/B test is one where the difference didn't reach significance.
It doesn't prove the variants are equal. It means the test couldn't detect an effect of the size it was able to measure.
This is the most common outcome: Optimizely's analysis of over 127,000 experiments found only about 12% won on their primary metric, and Spotify reports a similar 12% win rate.
Our database has 165 flat results. Three patterns show up again and again.
Big lifts on small samples. Clean Commit moved a "Made in the USA" flag above the product title on a Shopify store and saw +53.0% conversion rate.
But the test had only 1,408 visitors over 38 days, so the agency labeled it a positive signal, not a win.
See the "Made in the USA" test. A huge lift on few visitors is more often noise than a discovery.
Metrics that disagree. In another Clean Commit test, a sticky mobile add-to-cart button raised add-to-cart 6.8% while conversion rate fell 7.7% and RPV fell 22.1%, across 7,948 mobile visitors.
Nothing was significant. See the mobile sticky add-to-cart test.
Similarly, restoring an older mobile menu lifted conversion 5.4% but cut RPV 13.8% and AOV 18.2% (8,086 visitors, 66.5% probability on conversion rate).
See the mobile navigation test. A second run of the same revert lost 5.7% on conversion.
When metrics point in opposite directions, the honest call is "no clear effect."
Contaminated tests. A Clean Commit homepage personalized for new vs. returning visitors showed +241.1% conversion rate, but overlapping tests and holiday promotions interfered, and it didn't reach significance. See the personalized homepage test.
Blend Commerce
Clean CommitClean CommitClean CommitClean CommitWhat to do next with a flat result:
- Check whether the test could ever have won. Compare the observed sample with the sample needed for your minimum detectable effect. If it was underpowered, the result says little either way.
- Report the interval. "Somewhere between -4% and +6%" is useful: it rules out a large effect.
- Decide on other grounds. If the change is cheaper to maintain or better for users and the interval rules out a meaningful loss, shipping it is a reasonable call. Say that's why.
- Make a bolder change. Small tweaks produce small effects that low-traffic sites can't detect.
- Log it. Spotify counts about 64% of its experiments as producing a clear learning, far more than the share that win. Flat tests are part of that.
For more on running the test itself, start at the A/B testing hub.