Simpson's paradox is when a trend that shows up in every segment of your data disappears or flips once you add the segments together.
In A/B testing it usually happens when the share of traffic going to each variant changes partway through a test, or differs between segments, so the pooled numbers compare different mixes of visitors.
A variant can win on Friday and win on Saturday and still look like a loser on the combined report.
The fix is rarely fancy statistics. Keep the traffic split fixed for the whole test, check that each segment got the split you planned, and when the split did change, analyze each stable period on its own or combine them with weights.
What Simpson's paradox is
The effect is named after Edward Simpson, whose 1951 paper "The Interpretation of Interaction in Contingency Tables" in the Journal of the Royal Statistical Society described it.
Simpson wasn't the first to notice it. Judea Pearl's history of the paradox points to Karl Pearson and colleagues in 1899 and Udny Yule in 1903, and notes that Blyth was the one who called it a "paradox" in 1972.
Nothing about the math is broken. A pooled conversion rate is a weighted average of the segment rates, and the weights are each segment's share of that arm's visitors.
If control and variant have different weights, the pooled comparison mixes up "which variant" with "which visitors."
Pearl's argument is that the paradox goes away once you ask what caused the difference in mix.
When a third variable drives both who gets a treatment and the outcome, you should look at the segments.
When it doesn't, the pooled number is the honest one.
In a properly randomized A/B test with a fixed split, segments like device or traffic source can't drive which variant someone sees.
That's why the paradox in A/B tests almost always comes from something the team did to the split.
Simpson's paradox examples
Berkeley graduate admissions
The best-known example comes from fall 1973 applications to the University of California, Berkeley.
Bickel, Hammel and O'Connell (Science, 1975) found that about 44% of 8,442 male applicants and 35% of 4,321 female applicants were admitted. That looked like bias against women.
Each department ran its own admissions, though. Department by department, the authors found no general bias against women.
Women had applied more often to departments that were harder for everyone to get into, so the pooled rate punished them for where they applied, not for being women.
Kidney stone treatment
The second classic is a 1986 study comparing kidney stone treatments.
Overall, Charig and colleagues reported success in 273 of 350 patients (78%) after open surgery and 289 of 350 (83%) after percutaneous nephrolithotomy (PCNL). So PCNL looks better.
Split by stone size, as Julious and Mullee showed in the BMJ in 1994, open surgery wins in both groups:
| Stone size | Open surgery | PCNL |
|---|---|---|
| Small | 81 / 87 = 93% | 234 / 270 = 87% |
| Large | 192 / 263 = 73% | 55 / 80 = 69% |
| All | 273 / 350 = 78% | 289 / 350 = 83% |
Doctors gave open surgery to more of the hard cases (large stones).
Stone size drove both the treatment choice and the outcome, so the pooled comparison is the misleading one.
This is the same kind of imbalance a broken traffic split creates in an A/B test.
How it shows up in A/B tests: the ramp-up trap
Microsoft's experimentation team listed this as a pitfall in "Seven Pitfalls to Avoid when Running Controlled Experiments on the Web" (Crook, Frasca, Kohavi and Longbotham, KDD 2009).
Their example: a site with 1 million visitors a day starts a test at 1% treatment on Friday, then ramps to 50% on Saturday.
Saturday happens to be a weaker day for everyone.
| Friday (99% / 1%) | Saturday (50% / 50%) | Pooled | |
|---|---|---|---|
| Control | 20,000 / 990,000 = 2.02% | 5,000 / 500,000 = 1.00% | 25,000 / 1,490,000 = 1.68% |
| Treatment | 230 / 10,000 = 2.30% | 6,000 / 500,000 = 1.20% | 6,230 / 510,000 = 1.22% |
(The paper rounds the pooled treatment rate to 1.20%; the counts give 1.22%.)
Treatment wins both days, yet the pooled report says it loses, 1.22% vs 1.68%.
Why: about 98% of treatment visitors came on the weak Saturday, against about 34% of control visitors.
The paper lists other ways the same trap appears in online tests:
- Some browsers are sampled into the test at higher rates than others.
- Countries run different splits (say, 1% treatment in the US and 50% in Canada) and the results are pooled.
- Top customers are held to 1% exposure while everyone else runs 50/50.
The paper's authors say they have "seen them happen multiple times in real life."
Testing tools warn about this too. Optimizely's documentation separates traffic distribution (the share sent to each variation) from traffic allocation (the share of all visitors in the test).
Changing distribution mid-test "invites Simpson's paradox," it says.
Changing allocation doesn't, because both arms change together.
Multi-armed bandits shift traffic between arms on purpose, so they face the same problem.
Any tool that does this has to correct for changing weights over time.
See multi-armed bandits for when that trade-off is worth it.
Segment results that disagree are usually not Simpson's paradox
Many CRO teams use "Simpson's paradox" for any test where mobile and desktop go different ways.
That is a different thing: a real difference in effect by segment, which statisticians call heterogeneity or an interaction effect.
In a randomized test with a fixed split, the overall result is roughly a weighted average of the segment results.
It can't point the opposite way from every segment.
Our database has several clean examples of segments disagreeing:
- Blend Commerce tested two-image product cards for Stone Creek Coffee: +20% conversion rate and +11% revenue per visitor on mobile, but -12% conversion rate and -7% RPV on desktop. It shipped on mobile only.
- In Blend Commerce's simplified homepage hero for an apparel brand, desktop conversion rose 26% and desktop revenue per user rose 27%, while mobile fell. It shipped on desktop only.
- Convertibles' press quote test for an Australian gin subscription brand earned an extra A$9,392/month on desktop and lost on mobile. Mobile kept the control.
None of these is a reversal caused by uneven mixing.
Each one is a variant that works for one audience and hurts another.
When the split is stable, the overall number still tells you the average effect.
The segment numbers tell you where it came from.
Blend CommerceBlend Commerce
ConvertiblesBlend CommerceBe careful when you act on a segment win.
Every extra segment you check is another chance at a false positive (see multiple comparisons).
Segment estimates from smaller samples also tend to shrink when retested (the winner's curse).
Optimizely's guidance is to ship on the overall effect and use segments to find ideas for the next test.
If you think a segment really behaves differently, the safe move is a new test aimed at that segment.
Segment-only reporting: the quiet cousin
A related problem is a result reported for one segment with no overall number.
The aggregate might tell a different story, and you can't check.
Our database flags these cases. Blend Commerce's exposed mobile search bar for PerTronix reports +45% AOV and +46% revenue for returning visitors, and our caveat notes the overall result isn't given.
VWO's Showpo product info test reports +1.33% purchases among new visitors only.
These may be real effects, but read them as segment findings, not site-wide wins.
The clean alternative is to target the segment before the test starts.
Shirtinator ran its tablet test on Android tablet visitors only, as a 50/50 split URL test.
It found +71.81% revenue and +32% orders, and there's no aggregate to argue with because the whole test was one segment.
Split URL tests like this one send each arm to a different page.
How to check a test for Simpson's paradox
- Check that the split never changed. Look at the test's change log. Any change to variant weights mid-test splits the data into separate periods.
- Run a sample ratio check per period and per key segment. If you planned 50/50 but mobile ran 60/40, the segments are mixed unevenly. A sample ratio mismatch test (chi-square) flags this. An uneven split is fine as long as it's constant. Anatta's tabbed buy box test ran 85/15 (34,730 control and 6,144 variant visitors) and passed its SRM check.
- Compare pooled and per-segment directions. If the pooled result points one way and every segment points the other, the weights are uneven. Don't trust the pooled number until you know why.
- Look at segment shares by arm. In each arm, what share is mobile, new visitors, or paid traffic? In a healthy test these shares match between arms within noise.
How to fix it: worked example
The Microsoft team gives three fixes. Their simplest, which they use themselves, is to throw away the ramp-up data.
In the example above, that means using Saturday only:
lift = 1.20% / 1.00% - 1 = +20%
To keep both days, combine the per-period rates with fixed weights.
Weight each period by its share of total visitors, the same weights for both arms.
Each day had 1 million visitors, so each weight is 0.5:
stratified control = 0.5 × 2.02% + 0.5 × 1.00% = 1.51%
stratified treatment = 0.5 × 2.30% + 0.5 × 1.20% = 1.75%
lift = 1.75% / 1.51% - 1 ≈ +15.9%
The general formula, for periods or segments s with weight w(s) summing to 1:
stratified rate = Σ w(s) × conversions(s) / visitors(s)
The third fix is a paired comparison across periods where the split was stable.
Whichever you pick, decide before you look at the results.
Choosing the method that gives the nicest number is its own bias.
| Situation | Use | Wrong choice when |
|---|---|---|
| Fixed split for the whole test | Pooled result | Never wrong for the average effect; still check SRM |
| Short ramp-up, then a stable split | Drop the ramp-up period | The ramp-up is a large share of the data or covers a different season |
| Split changed for a long stretch | Stratified (weighted) estimate | Some periods have so few treatment visitors that their rates are mostly noise |
| Segments run at different splits by design | Stratify by segment | You use the per-segment numbers to pick winners without correcting for multiple comparisons |
| Segments disagree under a fixed split | Pooled result to ship; new targeted test for the segment | You call it Simpson's paradox and throw out the overall result |
Prevention is cheaper than any of these fixes.
Ramp by changing allocation rather than distribution, restart the test after any change to the split, and check sample ratios before reading results.
For the full checklist, see how to analyze A/B test results.
Want to see how often segment-specific wins show up?
Browse all tests in the database or start at the A/B testing hub.