Sample ratio mismatch (SRM) happens when the traffic split you get in an A/B test is meaningfully different from the split you set up, for example 50.6% / 49.4% on nearly 100,000 visitors when you asked for 50/50. A chi-square test tells you whether the gap is too large to be chance.
If it is, the test's results can't be trusted until you find the cause, because whatever pushed visitors out of one group may also have changed who was left.
SRM is the most common data-quality check in experimentation, and it's not rare.
Microsoft found it in about 6% of experiments and LinkedIn in about 10% of triggered experiments, according to the SRM Checker FAQ maintained by Lukas Vermeer, a co-author of the main research paper on the topic.
What an SRM test is and why it matters
An SRM test compares how many users each variant actually got with how many it should have got.
It doesn't look at conversions at all. It only asks one question: is this split plausible if the randomizer worked?
The reason it matters is selection. If a slow-loading variant loses visitors before its tracking fires, the visitors who do get counted are the patient ones.
Those visitors convert better, so the variant can look like a winner purely because the bounce-prone people disappeared from its denominator.
That's why a mismatch isn't something you can "adjust for" by reweighting.
The missing users are missing for a reason, and that reason is usually linked to the outcome you're measuring.
Where SRMs come from
The best-known study is "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments" by Aleksander Fabijan, Lukas Vermeer, Pavel Dmitriev and colleagues (KDD 2019).
It draws on experiments across four companies and more than 25 products, and sorts SRM causes into a taxonomy.
In practice, most causes fall into a few buckets:
| Stage | What goes wrong | Typical example |
|---|---|---|
| Assignment | Users aren't split the way you think | Bucketing bug, changing the split mid-test, overlapping experiments that share users |
| Execution | One variant drops or adds users after assignment | Redirect in a split-URL test, slower page, JavaScript error, variant breaks for one browser |
| Logging and tracking | Users are assigned but not counted equally | Analytics tag fires later on the variant, consent banner blocks tracking unevenly, bot traffic hits one URL |
| Analysis | The counting itself is off | Wrong date range, filtering on a metric the variant changed, counting sessions instead of users |
Execution and tracking problems are the ones most stores run into.
Any test where the variant loads differently from the control is a candidate, which is why split-URL tests and theme or page-builder tests deserve an SRM check before anyone reads the lift.
Tests in our database where the split deserves a look
None of the tests below reported an SRM.
They are listed because their setup is the kind where you'd want to confirm the split before trusting the number.
- Rebuilt a landing page natively in Shopify vs the same page in a page builder: Oliver Kenyon reported +20.9% revenue per visitor, +9.9% conversion rate and +10.0% AOV. The only change was the build, so speed is the treatment, and speed is also a classic SRM cause if tracking fires after the page loads.
- Moved a $600+ AOV offer page from SamCart to PageDeck: Peter Christensen reported +20% conversion rate. Moving between platforms usually means a redirect or a different tracking setup per arm.
- Non-blocking cookie banner instead of a consent wall: Blend Commerce reported +32% conversion. When the treatment is the consent flow itself, the share of visitors your analytics can see may differ between arms, so check the split with server-side or platform counts, not consent-gated analytics.
- New theme with a BOGO offer vs old theme: Kasper Dolk reported revenue per visitor going from £1.08 to £1.54 (+41.85%). Whole-theme tests change load time, scripts and URLs at once.
@oliverkenyon
Vasta
Blend Commerce
@kasper_dolk@kasper_dolk
@peterac_The good habit to copy is reporting visitors per arm.
In upsells vs no upsells, Kasper Dolk published roughly 398.6k visitors per arm alongside revenue per visitor of $1.46 vs $1.49 (+1.86%).
With arm counts published, anyone can run the SRM check themselves.
How to calculate sample ratio mismatch (calculator method)
The standard method is Pearson's chi-square goodness-of-fit test on the count of users (not sessions or pageviews) in each variant.
Kameleoon's SRM documentation and the SRM Checker both use it.
You can run it in our SRM checker or by hand.
The formula
- Expected count for each variant = total users × the share you assigned to it.
- For each variant: (observed − expected)² ÷ expected.
- Add those up. That sum is the chi-square statistic.
- Look up the p-value with degrees of freedom = number of variants − 1 (1 for a simple A/B test).
In a spreadsheet: =CHISQ.TEST(observed_range, expected_range) returns the p-value directly. In Python: scipy.stats.chisquare(observed, expected).
Worked example
You set a 50/50 split. After two weeks, control has 50,000 users and the variant has 48,800, for a total of 98,800.
| Control | Variant | |
|---|---|---|
| Observed users | 50,000 | 48,800 |
| Expected users (98,800 × 0.5) | 49,400 | 49,400 |
| Difference | +600 | −600 |
| (Difference)² ÷ expected | 360,000 ÷ 49,400 = 7.29 | 7.29 |
Chi-square = 7.29 + 7.29 = 14.57, with 1 degree of freedom.
The p-value is about 0.00013. That's below any common SRM threshold, so this test has an SRM, even though the split (50.6% / 49.4%) looks close to even.
Why percentages alone mislead
The same 51% / 49% split means very different things at different sample sizes:
| Control | Variant | Split | Chi-square | p-value |
|---|---|---|---|---|
| 5,100 | 4,900 | 51.0% / 49.0% | 4.0 | 0.046 |
| 10,200 | 9,800 | 51.0% / 49.0% | 8.0 | 0.0047 |
| 20,000 | 19,700 | 50.4% / 49.6% | 2.27 | 0.13 |
| 50,000 | 48,800 | 50.6% / 49.4% | 14.57 | 0.00013 |
Uneven splits work the same way: for a 90/10 test, multiply the total by 0.9 and 0.1 to get the expected counts.
Something like 9,150 vs 850 out of 10,000 (expected 9,000 vs 1,000) gives chi-square = 25 and p ≈ 0.0000006, a clear SRM.
Which threshold to use
SRM checks use a much stricter threshold than the usual 0.05 for statistical significance.
You run the check on every test, and a 0.05 cutoff would raise false alarms on 1 in 20 healthy tests.
- p < 0.001: the cutoff Kameleoon documents, and a sensible default.
- p < 0.01: what the SRM Checker Chrome extension used. It's more sensitive, so expect more borderline flags.
- Sequential checks: Optimizely's automatic SRM detection uses a sequential Bayesian multinomial test that checks daily and starts after more than 1,000 visitors, so you can peek without inflating false alarms.
In the table above, the 10,200 / 9,800 test is flagged at 0.01 but not at 0.001. Treat results in that zone as "investigate" rather than "throw away."
When the chi-square check is the wrong tool
- Counting sessions or pageviews. These aren't independent: one user can create many. Run the check on users (or whatever unit you randomized on), or the test will flag problems that aren't there.
- Repeated peeking with a fixed threshold. Checking a plain chi-square test every day raises the chance of a false alarm. Use a stricter cutoff or a sequential method if you monitor continuously.
- Very small samples. With a few hundred users the test has little power to spot a real SRM, and expected counts under about 5 per group break the chi-square approximation. A clean result early on doesn't prove the split is healthy.
- Segments. A test can pass overall and fail within one browser or device. Check key segments too, keeping in mind that more checks mean more false alarms.
SRM in A/B testing: how to diagnose and fix it
When the check fails, don't read the result, and don't ship the variant.
Work through the causes in roughly this order, because the cheap checks catch most problems.
- Confirm the counts. Make sure both arms use the same date range, the same unit (users), and the same filters. Analysis mistakes are the easiest SRMs to fix.
- Check whether the split changed. Did anyone change the traffic allocation, pause a variant, or restart the test? Changing allocation mid-test is a common assignment cause.
- Slice by segment and by day. If the gap appears only on Safari, only on mobile, or only after a release date, you've found your lead.
- Compare load and tracking. Does the variant redirect, load slower, or fire its tracking later? Does a consent banner or ad blocker affect one arm more? This is where page-builder, theme and client-side tests usually fail.
- Look for bots and internal traffic. A crawler or QA script hitting one URL can tilt the split.
- Fix, then restart. Once you've found the cause, fix it and rerun the test with fresh data. Trimming the data until the split looks even only hides the bias.
How to prevent it
- Run an A/A test when you set up a new tool or tracking. It should pass the SRM check, and if it doesn't, the problem is in your setup, not your ideas.
- Make the SRM check part of test QA and of your results analysis, alongside your guardrail metrics.
- Reduce the flicker and load difference between arms, and fire assignment tracking as early as possible.
- Keep the split fixed for the whole test. If you need to ramp up, analyze each allocation period separately.
- Publish users per arm with every result, as the upsells test above does.
SRM is one of the most common A/B testing mistakes because it's invisible unless you check.
Browse all tests in the database or the wider A/B testing guide.