Twyman's law says that any figure that looks interesting or different is usually wrong.
In A/B testing, it means a surprisingly large lift should be treated as a probable bug or fluke until proven otherwise.
The winner's curse explains why even honest big wins shrink: when you only ship results that cleared significance, the lift you report is, on average, larger than the real effect.
This page covers both ideas, shows the math behind exaggerated lifts, and uses real tests from our database where big early numbers faded, failed a rerun, or held up.
Twyman's law: where it comes from and what it means
The law is named after Tony Twyman, a UK media and market researcher.
It is often stated as "the more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another" (summary and references).
Ron Kohavi and colleagues brought it into online experimentation.
In their KDD 2014 paper Seven Rules of Thumb for Web Site Experimenters, they write that when results are exceptionally strong, "we learned to call out Twyman's law."
Their reasoning is Bayesian: across thousands of experiments, few moved their north-star metric much, so a huge jump is far more likely to come from an instrumentation error than from a breakthrough.
Their example: if past experiments on sessions per user cluster around zero with a standard deviation of 0.25%, a +2.0% result sits eight standard deviations out.
Even if it is statistically significant, the team starts hunting for the bug instead of celebrating.
The law is chapter 3 of their book, Trustworthy Online Controlled Experiments (Kohavi, Tang and Xu, 2020).
Twyman's law is not a claim that big wins never happen.
It is a rule about where to put your effort: the more surprising the number, the more checking it deserves before anyone ships it or puts it in a slide deck.
The winner's curse in A/B testing
The winner's curse is a selection effect. Every test result is the true effect plus noise.
If you only declare winners when the observed lift clears a significance threshold, you mostly keep results where the noise happened to push upward.
The kept estimates are biased high, even when the test was run correctly.
Minyong Lee and Milan Shen at Airbnb described this for experimentation programs in "Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments" (KDD 2018).
They showed that adding up the measured lifts of launched features overstates their total impact, and proposed a correction that they implemented in Airbnb's experiment reporting framework.
The curse gets worse as power drops. Kohavi, Deng and Vermeer put it plainly in A/B Testing Intuition Busters (KDD 2022): the "lucky" experimenter who finds an effect in a low-power setting, or through repeated tests, "is cursed by finding an inflated effect."
They also cite Button et al. (2013), who estimated that in neuroscience studies with power of 8% to 31%, initial effects were inflated by 25% to 50%.
The same paper invokes Twyman's law on a shared test that claimed a lift of over 300% to overall conversion.
The authors note they had been involved in tens of thousands of A/B tests at Airbnb, Booking, Amazon and Microsoft and had never seen a change improve conversion anywhere near that much.
They estimated its pre-experiment power at about 3%.
Two tests in our database that show the curse at work
Conversion Fanatics ran a mobile test that expanded all product options by default.
It led by more than 40% early on, then faded to a loss of about 4% in conversions.
Conversion Fanatics recorded it as a cautionary example of early results fading.
Convertibles saw a softer version in a cart drawer test with free shipping and pack upgrades.
The final read was +24% revenue per visitor on 119 vs 94 orders, with an 88% probability to beat control.
Convertibles noted that earlier reads showed a bigger lift that shrank.
Both are textbook cases. Early in a test, samples are small, noise is large, and whichever arm got lucky looks like a big winner.
If either test had been stopped at its peak, the reported lift would have been the noise, not the effect.
Why exaggerated lifts happen, with numbers
Andrew Gelman and John Carlin gave this a name in "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" (Perspectives on Psychological Science, 2014).
The exaggeration ratio, or Type M error, is the expected size of a significant estimate divided by the true effect.
The Type S error rate is the chance that a significant result points the wrong way.
They show that at 80% power the exaggeration ratio is about 1.12, which is small.
Problems start below 50% power, and the sign itself becomes unreliable below about 10% power.
In one of their published-study examples, power was 0.06, the Type S error rate 24%, and the expected exaggeration 9.7 times.
Exaggeration by power level
The table below is our own simulation using a normal approximation and a two-sided test at α = 0.05, following Gelman and Carlin's method.
"True effect" is measured in standard errors of the difference.
| Power | True effect (standard errors) | Average significant estimate ÷ true effect | Share of significant results with the wrong sign |
|---|---|---|---|
| 80% | 2.80 | about 1.13x | under 0.01% |
| 50% | 1.96 | about 1.4x | about 0.01% |
| 30% | 1.43 | about 1.8x | about 0.1% |
| 20% | 1.11 | about 2.3x | about 0.6% |
| 10% | 0.65 | about 3.7x | about 4.6% |
Worked example: a small store testing a modest change
Say a store converts at 3% and runs a 50/50 test with 2,000 visitors per arm.
The real effect of the change is +10% relative, so 3.0% becomes 3.3%.
- Standard error of the difference: √(2 × 0.03 × 0.97 ÷ 2,000) ≈ 0.54 percentage points.
- To be significant at 95% (two-sided), the observed gap must exceed 1.96 × 0.54 ≈ 1.06 points. On a 3% baseline that is a relative lift of about +35%.
- The true gap is 0.3 points, about 0.56 standard errors. Power is roughly 9%.
So this test can only "win" by reporting a lift of +35% or more, when the truth is +10%.
In our simulation, the average significant positive result came out at about +43%, more than four times the real effect, and about 7% of significant results pointed the wrong way.
Nothing went wrong in the setup. Selection on significance did all of it.
This is why a significant result from an underpowered test is not "a smaller win than it looks but still a win."
The size is unreliable, and sometimes the direction is too.
Check power before you start with a sample size calculator, and see our guide to statistical power for the details.
What exaggeration looks like in our database
Of the 1,724 results in LiftArchive, 1,447 are recorded as wins.
Of the wins with a positive percentage in the headline, the median is +18%, 271 report +50% or more, and 129 report +100% or more.
Those headline figures mix metrics (clicks, sign-ups, orders, revenue), and many come from published case studies, which tend to feature winners.
Read our biggest A/B test wins with Twyman's law in mind.
Some large numbers are real but narrow. VMware's test moving try and buy CTAs above the fold reported +956% clickthrough on an SMB solutions page where the CTAs had been buried among links at the bottom.
A click metric on a previously hidden button can move that much.
That does not mean revenue moved by anything like the same amount.
Clean Commit's free-shipping eligibility message on product pages shows the low-power case.
It reported +21.0% conversion rate and +31.7% revenue per visitor on 4,128 visitors over 26 days, with 84.6% probability and 15% power.
Clean Commit labeled it a directional "positive signal" and noted to expect shrinkage on a retest.
That is the honest way to report it.
Convertibles
Clean Commit
Conversion Fanatics
MarketingExperimentsHow to check a big win before you ship it
Twyman's law tells you to look harder. Here is what looking harder means in practice.
- Check the plumbing first. Look for a sample ratio mismatch, broken tracking in one arm, bots, or a metric that changed definition mid-test. Kohavi's team found that extreme results usually turned out to be bugs, such as instrumentation errors.
- Compare the control to its own history. If the control converted far below its normal rate, the "lift" may be a control problem, not a variant success.
- Ask whether the metric could plausibly move that much. A +400% click-through on a newly visible button is believable. A +400% in revenue from a headline change is not.
- Check the power you actually had. If the test was underpowered for a realistic effect, treat the size of the lift as unknown and the direction as tentative.
- Rerun it, or hold back a group. A replication is the cleanest test of a surprising result. Expect the second number to be smaller, which is also what regression to the mean predicts.
- Watch for novelty. Some lifts are real at first and then decay as returning visitors get used to the change. See the novelty effect.
- Use a stricter bar for surprising claims. Intuition Busters recommends accepting a highly surprising result only if the p-value is very low.
A rerun that failed, and one that held
MarketingExperiments (MECLABS) ran a value proposition test on a tech supplier's top-selling product page.
One treatment showed +445.6% at 99% confidence. But the site got only 2 to 10 conversions per day, the control had six straight days with no conversions, and the control ran far below its 3.83% historical rate.
A rerun with a cloned control showed a 30% gap between identical pages, so the team did not trust the lift and moved to a larger template test instead.
Voices.com tested removing featured talent profiles from its homepage.
The first run showed +33% sign-ups, and a rerun to validate it showed +28%.
The effect shrank a little, as the winner's curse predicts, but held up.
That is what a real win usually looks like on a second run: similar direction, somewhat smaller.
When a rerun isn't possible, plan on a discounted forecast.
If the test was underpowered, assume the true lift is a fraction of what you measured, and say so when you report it.
More on running trustworthy tests is in our A/B testing guide.