A confidence level is the long-run success rate of the method you choose before the test starts, such as 95%.
A confidence interval is the range you calculate from your data after the test, such as "+1% to +30% lift."
The level says how often intervals built this way contain the true effect; the interval says which effects your test can't rule out.
In A/B testing, the interval is the more useful of the two.
A p-value or "chance to beat control" tells you whether there is probably some effect.
The interval tells you how big it might be, and whether the low end is still worth shipping.
What a confidence interval tells you in an A/B test
Every A/B test result is an estimate. If your variant converted at 5.46% and control at 4.73%, the observed lift is +15.6%, but a rerun with fresh visitors would give a different number.
The confidence interval puts bounds on how far off that estimate could reasonably be.
Take a real test from our database. Convertibles added a star rating under the price plus member reviews on a whisky subscription product page.
Conversion went from 4.725% to 5.460% (+15.56%) across 15,615 visitors and 795 orders, with a 98% probability of being best.
Assuming a roughly even split, the 95% confidence interval for that lift runs from about +1% to +30%.
That is our calculation from the published numbers, not a figure Convertibles reported.
It says the change very likely helped, but the true effect could be anywhere from tiny to large.
Planning revenue on "+15.6%" alone would overstate how much the test actually pinned down.
Three readings matter in practice:
- The interval excludes zero: the result is statistically significant at that level. This is the same decision a p-value below 0.05 gives for a 95% interval.
- The interval includes zero: the data can't rule out no effect. It also can't rule out a meaningful win or loss, which is why "not significant" is not the same as "no effect."
- The width: a narrow interval means a precise estimate. A wide one means you need more data before the exact size of the lift means much.
How to calculate a confidence interval for a conversion rate difference
For a standard two-variant conversion test, the most common method is the normal-approximation (Wald) interval for the difference between two proportions:
| Step | Formula (plain text) |
|---|---|
| Conversion rates | p1 = conversions_A / visitors_A, p2 = conversions_B / visitors_B |
| Difference | d = p2 - p1 |
| Standard error | SE = sqrt( p1(1 - p1)/n1 + p2(1 - p2)/n2 ) |
| Interval | d ± z × SE |
| z values | 1.645 for 90%, 1.96 for 95%, 2.576 for 99% |
| Rough relative lift | divide both ends by p1 |
Worked example
Using the whisky product page test above, with about 7,808 visitors per arm:
- p1 = 0.04725 and p2 = 0.05460, so d = 0.00735 (0.735 percentage points).
- SE = sqrt(0.04725 × 0.95275 / 7,808 + 0.05460 × 0.94540 / 7,808) = 0.00352.
- 95% interval: 0.00735 ± 1.96 × 0.00352 = 0.00735 ± 0.00690, so +0.045 to +1.425 percentage points.
- Divide by the control rate (0.04725): roughly +1.0% to +30.2% relative lift.
- At 90%, the margin shrinks to ± 0.00579, giving about +3.3% to +27.8%.
Dividing by p1 treats the control rate as fixed, which slightly understates uncertainty.
Tools that report relative lift intervals usually use a more careful method, so their numbers can differ a little from this shortcut.
You can run your own numbers through our A/B test significance calculator.
When this formula is the wrong choice
- Small samples or very low conversion rates. The Wald interval's coverage is erratic. Brown, Cai and DasGupta (2001) showed the problem persists well beyond the sample sizes textbooks call safe, and recommended the Wilson or Jeffreys interval for small n and Agresti-Coull for larger n.
- Peeking. A fixed-sample interval is only valid if you look once, at the planned sample size. If you check daily and stop when it excludes zero, real coverage drops well below 95%. Use a sequential method instead.
- Revenue per visitor and AOV. These are not proportions. Use a t-based interval on per-visitor values, or a bootstrap when a few large orders dominate.
- Many variants or metrics. With many 95% intervals, the chance that at least one misses its true value climbs quickly, so correct for multiple comparisons.
Confidence interval: what it means and what it doesn't
The confidence interval comes from Jerzy Neyman, who set out the theory in a 1937 paper "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability" in the Philosophical Transactions of the Royal Society (background on the method).
Neyman built intervals so that the procedure covers the true value with at least a chosen probability, and deliberately named them "confidence" intervals to avoid the word probability.
That distinction is where most people trip. A 95% interval of +1% to +30% does not mean there is a 95% probability the true lift is in that range.
It means that if you ran many identical tests and built an interval each time, about 95% of those intervals would contain the true lift. Any single interval either does or doesn't.
This confuses experts too. In Hoekstra, Morey, Rouder and Wagenmakers (2014), 120 psychology researchers and 442 students rated six false statements about a confidence interval, and both groups endorsed more than three on average.
Researchers barely outperformed students. A later replication by García-Pérez and Alcalá-Quintana (2016) argued the original items partly measured lack of knowledge and response bias rather than firm misconceptions, so treat the exact rates with some caution.
If you want a direct "there's a 95% chance the lift is between X and Y" statement, that is a Bayesian credible interval.
Many ecommerce tools report Bayesian "probability to beat control" for this reason.
See Bayesian vs frequentist testing for the trade-offs.
"Flat" results still have an interval
Blend Commerce tested value props below Add to Cart for a travel brand.
Conversion was flat at 3.30% in both arms across 99,754 visitors, and it was not significant.
A flat point estimate on that traffic still leaves a 95% interval of roughly -6.7% to +6.7% relative (our calculation, assuming an even split).
So the test rules out big effects, but not a 5% swing either way.
"No difference" really means "no difference larger than this test could detect."
Your minimum detectable effect sets that floor before the test starts.
GrowthRock shows the other side. Its 'You Save #%' test had 280,000 visitors per variation and all three versions landed within 0.3% of each other, a much tighter flat.
A follow-up with varied discounts, with 700,000 visitors and 18,000 conversions, detected a +2.57% conversion lift at 99% significance.
Small, real lifts only show up in narrow intervals, and narrow intervals take traffic.
Confidence level: choosing 90%, 95% or 99%
The confidence level is a setting you choose before the test.
It is 1 minus your significance threshold (alpha): a 95% level matches alpha = 0.05. Higher levels give wider intervals and fewer false positives, but you need more traffic to reach a conclusion.
| Level | z (two-sided) | False-positive rate when there's no effect | Width relative to 95% |
|---|---|---|---|
| 90% | 1.645 | 10% | about 16% narrower |
| 95% | 1.96 | 5% | baseline |
| 99% | 2.576 | 1% | about 31% wider |
Defaults vary by tool. Optimizely's documentation says its interval uses the same level as the project's significance setting, 90% by default.
It also shows a running intersection of intervals across the experiment, which keeps the interval valid when you check results often.
The level can flip your verdict
Convertibles' sticky mobile CTA test on a whisky subscription homepage moved conversion from 1.135% to 1.366% (+20.4%) over 25,935 visitors and 327 orders, with a 95.1% probability to beat control.
By our calculation, the 90% interval is about +0.4% to +40.4% and excludes zero.
The two-sided 95% interval is about -3.5% to +44.2% and includes it.
The same data is "significant" at one level and not at the other.
That is why you pick the level before launch and don't change it after seeing results.
A 95.1% one-sided probability lines up with the 90% two-sided interval, because a two-sided 90% interval puts 5% in each tail.
The same team's cart drawer test for high-ticket equipment went from 0.382% to 0.444% (+16%) on 84,005 visitors but only 347 orders, at 86-92% probability to beat control.
Its 95% interval is roughly -6.5% to +38.9%.
At very low conversion rates, order count, not visitor count, sets your precision.
How to report a lift with its interval
A result reported as "+24% conversion" sounds like a fact.
Reported as "+24%, 95% CI -10% to +60%", it reads as what it is: a hint.
Clean Commit's shorter mobile product gallery showed +24.0% conversion on 4,788 visitors with 95.3% probability to beat control but only 23% power, and the agency labeled it a positive signal rather than a win.
- Lead with the interval, not just the point estimate. "Between +1% and +30%" is more honest than "+15.6%."
- Plan on the low end. If the change is only worth shipping at +5%, and the lower bound is +1%, you have a decision to make, not a sure win.
- Expect winners to shrink. Tests that just cleared the bar tend to overestimate the effect. The winner's curse explains why reruns usually come in lower.
- State the level and method. "90% sequential" and "95% fixed-sample" intervals aren't comparable.
Browse every test in the database for more examples with visitor and order counts.
GrowthRock
ConvertiblesConvertiblesConvertibles
Blend Commerce
Clean CommitMore on test statistics: statistical significance and the A/B testing hub.