An SRM calculator checks whether the visitors in your A/B test split the way you configured them.
It runs a chi-square goodness-of-fit test on the visitor count in each variation.
If the p-value falls below 0.001, the split is too lopsided to be chance, and the test's results shouldn't be trusted until you find the cause.
Enter visitors per variation and your planned split in the calculator above.
The rest of this page covers the formula, a worked example using real numbers from our database, and the mistakes that lead to wrong answers.
What the sample ratio mismatch checker calculates
Sample ratio mismatch (SRM) happens when the share of users in each group doesn't match the share you asked for.
Say you set a 50/50 split and end up with 52/48 on 100,000 visitors.
That gap is much too large to be bad luck.
It is a common problem. In the KDD 2019 paper by Fabijan, Vermeer and colleagues, about 6% of experiments at Microsoft showed an SRM over one year, and the same paper cites LinkedIn work finding SRMs in about 10% of triggered analyses.
Our longer guide to sample ratio mismatch covers the causes in depth.
The checker answers one question: if assignment really were random at your planned ratio, how likely is a split at least this uneven?
It doesn't tell you whether your variant won.
For that, use the A/B test significance calculator, and run it only after the split passes.
The chi-square formula
The calculator uses Pearson's chi-square goodness-of-fit test. This is the same test documented by Kameleoon and Harness for their built-in SRM checks.
| Step | Formula |
|---|---|
| 1. Total visitors | N = sum of observed visitors across all variations |
| 2. Expected count per variation | E = N x planned share (for example N x 0.5) |
| 3. Chi-square statistic | X2 = sum over variations of (Observed - Expected)^2 / Expected |
| 4. Degrees of freedom | df = number of variations - 1 |
| 5. p-value | probability of a chi-square value at least X2 with df degrees of freedom |
Inputs: the visitor (or user) count assigned to each variation, and the planned traffic share for each.
Shares must add to 100%. Count the unit you randomize on: if the tool buckets users, enter users, not sessions or pageviews.
Worked example: a real 85/15 test and a broken 50/50 test
Anatta ran a tabbed buy box with a "Best Value" bundle tab on an 85/15 split.
It reported 34,730 visitors in control and 6,144 in the variant, passed an SRM check, and lifted AOV by about 2%. Here is the math:
| Control | Variant | |
|---|---|---|
| Observed | 34,730 | 6,144 |
| Planned share | 85% | 15% |
| Expected (40,874 x share) | 34,742.9 | 6,131.1 |
| (O - E)^2 / E | 0.0048 | 0.0271 |
X2 = 0.032 with 1 degree of freedom, so p is about 0.86. A split at least this uneven would happen by chance most of the time, so there's no sign of SRM. Anatta's own check agreed.
Now a broken example. A 50/50 test shows 50,000 visitors in control and 48,900 in the variant.
Each expected count is 49,450, so X2 = (550^2 / 49,450) x 2 = 12.23, and p is about 0.0005. The split is only 50.6/49.4, but on this many visitors it's a clear SRM.
Equal counts in a write-up aren't proof either.
GrowthHit's sitewide free-shipping banner reported 26,306 visitors per arm, and GrowthRock's "You Save #%" price test ran about 280,000 visitors per variation.
Rounded or averaged figures like these can hide a mismatch, so always check the raw counts from your testing tool.
SplitBase
GrowthRockHow to read the result
- p below 0.001: SRM detected. Kameleoon and Harness both flag at this threshold. Don't ship and don't report the lift. Find the cause first.
- p between 0.001 and 0.05: watch it. At these thresholds, healthy tests rarely trip a false alarm (about 1 in 1,000 at 0.001). A borderline value is worth a look at segments and dates, but on its own it doesn't prove a problem.
- p above 0.05: no evidence of SRM. This doesn't prove the setup is clean. A redirect bug that drops a few hundred users can hide in a small test.
A strict cutoff is used on purpose. Teams check SRM on every test and often every day, so a 0.05 cutoff would raise false alarms constantly.
Chi-square calculator for A/B tests with more than two variations
The same formula works for A/B/n tests: add one row per variation, and degrees of freedom become variations minus 1. SplitBase's professional headline test for BestSelf ran a control plus two challengers at about 20,000 visitors each, which is the kind of setup you'd check this way.
Example: 10,200 / 10,050 / 9,750 visitors on an even three-way split.
The expected count is 10,000 each, so X2 = 4 + 0.25 + 6.25 = 10.5 with 2 degrees of freedom, and p is about 0.005. That's suspicious but under the 0.001 bar, so look at the 9,750 arm first.
More on running several arms in A/B/n testing.
Common mistakes
- Checking conversions instead of visitors. Orders are supposed to differ between arms. That's the effect you're measuring. Check only the assignment unit.
- Checking a filtered segment. If the variant changes who reaches a page, a "reached checkout" segment will mismatch even when assignment is fine. Harness lists event-based qualification as a common cause.
- Redirecting only one arm. Split-URL tests lose users in the redirect. The KDD paper recommends redirecting both control and variant when one needs it.
- Entering the wrong planned split. If you ramped traffic from 10% to 50% partway through, the overall ratio is a blend. Check each ramp stage separately.
- Re-running analysis after an SRM. Dropping "bad" users after the fact rarely fixes it. Find the bug (bots, flicker, script errors, caching), fix it, and restart. An A/A test and proper test QA catch most setup issues before launch, and flicker is a frequent source of client-side losses.
Once the split passes, move on to analyzing your results, or browse more real A/B test results and the wider A/B testing guides.