Use a two-proportion z-test (or the equivalent chi-square test) when your metric is a conversion rate, and a Welch t-test when your metric is an average like revenue per visitor or order value.
In A/B tests with thousands of visitors, the z-test and t-test give almost the same answer, because the t-distribution becomes the normal distribution as the sample grows.
The choice that matters is what kind of metric you are testing, not z versus t.
Here is the short version, then the detail behind each choice, a worked example using a real test from our database, and the cases where each test breaks.
| Your metric | Example | Test to use | Wrong choice when |
|---|---|---|---|
| Yes/no per visitor (rate) | Conversion rate, add-to-cart rate | Two-proportion z-test or chi-square test (same result for two arms) | Any expected cell count is under 5: use Fisher's exact test |
| Rate across 3+ variants | A/B/C test on conversion | Chi-square test of independence | You need to know which variant won: follow up with pairwise tests and a correction |
| Average per visitor or order | Revenue per visitor, AOV | Welch t-test | Very skewed data with too few visitors: cap outliers or bootstrap |
| Traffic split check | 50/50 split came out 51/49 | Chi-square goodness of fit | Never skip it |
Chi-square vs t-test (and where the z-test fits)
A chi-square test compares counts. A t-test compares means.
That is the whole difference, and it maps neatly onto the two kinds of metrics in most A/B tests.
If every visitor either converted or didn't, your data is a 2x2 table of counts: converted and not converted, for control and variant.
The chi-square test asks whether conversion depends on which arm the visitor saw.
The two-proportion z-test asks the same question in a different form, and for a 2x2 table the chi-square statistic is exactly the z statistic squared, so the two-sided p-values are identical.
Penn State's STAT 500 notes on the 2x2 table show the two side by side.
If each visitor produces a number, such as dollars spent (zero for most visitors), you are comparing two averages.
That is t-test territory. You can't run a chi-square test on revenue without throwing information away by binning it.
Z-test vs t-test
The z-test assumes you know the spread of the data.
The t-test estimates the spread from the sample and pays for that uncertainty with slightly wider critical values.
With 10 observations per group (18 degrees of freedom) the gap is real: the two-sided 5% cutoff is about 2.10 for t versus 1.96 for z.
With 10,000 visitors per arm the t cutoff is 1.960 to three decimals, and the tests agree.
For a conversion rate, the "known spread" is not really an assumption.
A rate p has variance p(1 - p), so once you estimate the rate you have the spread.
That is why the z-test is the textbook choice for proportions. The NIST/SEMATECH e-Handbook gives the large-sample formula:
z = (p1 - p2) / sqrt( p * (1 - p) * (1/n1 + 1/n2) ), where p = (x1 + x2) / (n1 + n2) is the pooled conversion rate.
For averages, you never know the true spread in advance, so you use the t-test.
In practice the z-versus-t question only matters for small samples, and small samples have bigger problems than the choice of test, such as too few conversions to detect anything.
Using a t-test in A/B testing
Use a t-test for any metric that is an average per unit: revenue per visitor, average order value, items per order, sessions per user.
The default should be Welch's version, from Welch's 1947 Biometrika paper, which does not assume the two groups have equal variance.
The NIST handbook's two-sample t-test page gives the statistic:
t = (mean1 - mean2) / sqrt( s1^2/n1 + s2^2/n2 )
Here s1 and s2 are the sample standard deviations.
A variant that changes buying behavior often changes the spread too, so the equal-variance Student t-test is the wrong default.
The problem with revenue: skew
The t-test relies on the average being roughly normally distributed.
Revenue per visitor is extremely skewed: most visitors spend nothing and a few spend a lot.
Kohavi, Deng, Longbotham and Xu's Seven Rules of Thumb for Web Site Experimenters (KDD 2014) gives a rule: you need at least 355 × s² observations per variant, where s is the skewness, before the average behaves normally.
They recommend applying it when skewness is above 1.
At Bing, revenue per user had a skewness of 17.9, which works out to about 114,000 users per variant.
Capping revenue at $10 per user per week cut the skewness to 5.2 and the requirement to about 9,700 users.
The same paper notes that an equal 50/50 split makes the difference between arms close to symmetric, which eases the problem.
That matters for tests like Anatta's tabbed buy box with a Best Value bundle tab, which raised AOV from $102.63 to $104.74 (about +2%).
It ran on an 85/15 split (34,730 control visitors, 6,144 variant), and Anatta filtered out wholesale orders, outliers and orders under $20 before analysis.
Filtering or capping like that is exactly the kind of step that makes a mean comparison on skewed data trustworthy.
Revenue needs more data than conversion
Because revenue carries the spread of order values on top of the yes/no of buying, the same test usually reaches significance on conversion before revenue.
Convertibles' sticky mobile CTA on a whisky subscription homepage lifted conversion from 1.135% to 1.366% (+20.4%) at a 95.1% probability to beat control, but revenue per visitor (+19.6%) was only at 87.3%.
Blend Commerce's value props below Add to Cart test ran on 99,754 visitors and saw about +2% revenue per visitor, still not significant.
Plan for this before launch: size the test for the revenue metric, not just conversion.
Our A/B testing guides cover sample size, power and test length.
When the t-test is the wrong choice
- Small, skewed samples. If you are below the 355 × s² line, cap extreme values or use a bootstrap, which the Kohavi paper also suggests.
- Swapping in Mann-Whitney "because the data isn't normal." The Mann-Whitney test does not compare means. Fagerland (2012) shows that when distributions differ in spread it tests whether a random visitor from one group tends to beat one from the other, and concludes that large studies should use t-tests even for heavily skewed data. Your business cares about mean revenue, so test the mean.
- Metrics where the unit isn't the visitor. Revenue per session, when visitors were randomized, breaks the independence assumption. Analyze per randomized visitor.
Using a chi-square test in A/B testing
Use a chi-square test for any yes/no metric, and especially when you have more than two variants.
It also powers the most important sanity check in experimentation.
Worked example: one real test, three tests, one answer
Convertibles' star rating under the price plus member reviews test on a whisky subscription product page moved conversion from 4.725% to 5.460% (+15.56%) on 15,615 visitors and 795 orders, with a 98% probability of being best.
Convertibles reported only the total visitors, so the per-arm numbers below are our calculation assuming an even split.
| Visitors | Orders | Not converted | Rate | |
|---|---|---|---|---|
| Control | 7,808 | 369 | 7,439 | 4.73% |
| Variant | 7,807 | 426 | 7,381 | 5.46% |
| Total | 15,615 | 795 | 14,820 | 5.09% |
- Pooled rate: 795 / 15,615 = 0.0509.
- Standard error: sqrt(0.0509 × 0.9491 × (1/7,808 + 1/7,807)) = 0.00352.
- Z-test: (0.05457 - 0.04726) / 0.00352 = 2.08. Two-sided p = 0.038.
- Chi-square test: expected orders per arm = 397.5. Summing (observed - expected)² / expected over all four cells gives 4.31, which is 2.08². With 1 degree of freedom, p = 0.038.
- Welch t-test on the 0/1 data: t = 2.08, df about 15,600, p = 0.038.
All three agree to two decimal places. A 95% confidence interval for the difference is about +0.04 to +1.42 percentage points, so the lift is real but its size is uncertain.
Our confidence intervals guide covers how to read that range, and the A/B test significance calculator runs this math for you.
A counter-example: low base rates
Convertibles' cart drawer instead of an add-to-cart banner on a high-ticket equipment store moved conversion from 0.382% to 0.444% (+16%) across 84,005 visitors and 347 orders, at 86-92% probability to beat control.
Assuming an even split, that is about 160 versus 186 orders, which gives z of about 1.40 and a two-sided p of about 0.16 by our calculation.
Big traffic doesn't help much when the conversions are rare: the order counts are what drive the test.
More than two variants
For an A/B/C test, a chi-square test on the 3x2 table (three arms, converted or not) tells you whether conversion differs anywhere across the arms.
It does not tell you which arm won.
Blend Commerce's subcategory pills on the homepage was an A/B/C test paused after 5,342 visitors, with the pill variant at +22.86% RPV and +13.54% conversion and an 83.1% probability to beat control.
Picking the best of two variants versus control means two comparisons, so correct for that as described in our multiple comparisons guide.
The sample ratio mismatch check
A chi-square goodness-of-fit test compares how many visitors each arm got against the split you planned.
A very small p-value means the randomization or tracking is broken and the result shouldn't be trusted.
Anatta's buy-box test above passed this check on its planned 85/15 split.
Read more in our sample ratio mismatch guide.
When the chi-square test is the wrong choice
- Small expected counts. The chi-square p-value is an approximation. The common rule of thumb is that every expected cell count should be above 5; below that, use Fisher's exact test (see The Analysis Factor's summary of the rule).
- Directional hypotheses. Chi-square is inherently two-sided. If you pre-registered a one-sided test, use the z-test. See one-tailed vs two-tailed tests.
- Peeking. None of these tests allow checking daily and stopping at the first significant result. If you need to monitor, use sequential testing.
Which test does your tool run?
Most testing platforms pick the test for you, and many no longer report a classic z-test or t-test at all.
Several of the results above came from Bayesian reports ("probability to beat control"), which answer a different question than a p-value.
Optimizely's Stats Engine whitepaper describes a sequential method with false discovery rate control rather than a fixed-sample t-test.
If you export raw data to check a result yourself, the rule stays simple: z or chi-square for rates, Welch t for averages, chi-square goodness of fit for the split.
Then interpret the p-value correctly and browse more real test results to see how others report theirs.
Convertibles
Blend CommerceConvertiblesConvertiblesBlend Commerce