Regression to the mean is the tendency for an unusually high or low measurement to be followed by one closer to the average, purely because part of the first measurement was luck. If you change a page right after a bad week and compare the next weeks to that bad week, some or all of the "lift" you see would have happened anyway.
A randomized A/B test, where control and variant run at the same time, is the standard way to separate a real effect from this drift.
What regression toward the mean means
"Regression to the mean" and "regression toward the mean" are the same idea.
Any number you measure, such as a week's conversion rate, has two parts: the stable underlying rate and noise from that week's particular mix of visitors.
When you pick a measurement because it is extreme, you have most likely picked one where the noise happened to push hard in one direction.
The next measurement gets fresh noise, which on average pushes neither way, so it lands closer to the true rate.
A 2005 review in the International Journal of Epidemiology by Barnett, van der Pols and Dobson describes it as a statistical effect that "can make natural variation in repeated data look like real change," and says it can be reduced through study design and suitable analysis.
Nothing about the page, the product or the customer has to change for it to happen.
Where the name comes from
Francis Galton described the pattern in 1886 in "Regression towards Mediocrity in Hereditary Stature", published in the Journal of the Anthropological Institute.
Galton found that children of very tall or very short parents tended to be less extreme than their parents.
"Mediocrity" meant the average, and the word "regression" in statistics comes from this work.
The classic mistake: praise and punishment
Daniel Kahneman's Nobel autobiography tells of an Israeli Air Force flight instructor who argued that praising cadets after a good maneuver made them do worse next time, while screaming at them after a bad one made them improve.
Kahneman realized the instructor was watching regression to the mean: extreme performances, good or bad, were followed by more ordinary ones regardless of the feedback.
Kahneman then had participants toss coins at a target without feedback.
Those who did best the first time mostly did worse on the second try, and the reverse.
CRO teams fall into the same trap when they act on a bad month and credit the change for the recovery.
How big the effect is: a worked example
The standard way to estimate it uses the correlation between one period and the next (call it r, from 0 to 1):
expected next value = long-run mean + r × (observed value − long-run mean)
If r is 1, periods are perfectly consistent and there is no regression.
If r is 0, the period-to-period differences are pure noise, and the expected next value is just the long-run mean.
Most store metrics sit somewhere in between, and smaller samples push r toward 0.
Example: one store, one bad week
Say a store converts at 2.0% over the long run and gets 1,000 sessions a week.
Sampling noise alone gives a weekly standard deviation of about 0.44 percentage points: sqrt(0.02 × 0.98 / 1,000) = 0.0044. A 1.4% week is only about 1.4 standard deviations below normal, which will happen by chance several times a year.
| Assumed week-to-week correlation (r) | Bad week | Expected next week with no change | Apparent "lift" |
|---|---|---|---|
| 0 (all noise) | 1.4% | 2.0% | +43% |
| 0.5 | 1.4% | 1.7% | +21% |
| 0.9 | 1.4% | 1.46% | +4% |
If the team ships a new hero section after that 1.4% week and next week comes in at 1.7%, a before/after report shows a 21% lift.
Under the middle assumption, that entire lift is regression.
Example: many stores, worst performers
We ran a simple simulation: 2,000 stores, every one with an identical true conversion rate of 2.0% and 1,000 sessions a week.
Nothing changed between week one and week two.
The 10% of stores with the worst week one averaged 1.25%.
In week two, the same stores averaged 2.01%, a 60% "improvement."
The best 10% fell from 2.80% to 2.01%.
This is the situation an agency faces when clients arrive after a slump.
The stores that ask for help are disproportionately the ones having an unlucky stretch, so their next period will tend to look better no matter who works on them.
Why before/after results are the riskiest evidence
A before/after comparison measures one period, makes a change, then measures a later period.
Regression to the mean is one of several things it cannot separate from the change itself, along with seasonality, traffic mix, promotions and ad spend.
Our methodology labels each entry by its type of evidence, and 83 of the 1,724 results in the database are tagged before/after for this reason.
Here is how those risks look in real entries.
We think several of these changes probably helped; the point is that the design of the comparison can't prove by how much.
- Venture Harbour's multi-step enquiry form went from 0.96% to 8.1% conversion. The source describes several years of ongoing iteration rather than one split test, so the starting point, the traffic and the form all changed over that time.
- Blend Commerce's Six Brothers Beef product page work reports +177% conversion rate, comparing June to August 2026 with the same months of 2025. Sessions fell 24% and AOV fell 30%, and the agency itself says the results also reflect wider trading conditions. Fewer, more committed visitors can raise conversion rate by themselves.
- Kasper Dolk's custom checkout blocks show a 3.8% conversion rate, up 24% month to date against the previous period. Sessions rose 58% at the same time, which means the visitor mix changed along with the checkout.
- Prismfly's Zoku theme redesign reports +15% conversion rate after launch, and the agency notes that sales were partly boosted by large B2B orders.
Not every before/after result is equally exposed. Oliver Kenyon's offer-focused hero headline took a brand from 0.9% to 3.6% conversion, and the 0.9% was described as the brand's average conversion rate.
A long-run average is much less likely to be a lucky low than a single bad week, so regression explains less of a jump like that.
A four-fold change is also far outside normal week-to-week noise.
@oliverkenyon
Prismfly
Vasta
@kasper_dolk
Blend Commerce
Venture HarbourHow a controlled A/B test removes it
In an A/B test, visitors are randomly split between control and variant during the same days.
If the store was in a slump and is now recovering, both groups recover together.
The difference between them is what the change did, plus random error that the statistics account for.
That is why a retest is so useful.
Convertibles reran a product page social proof test for an Australian whisky subscription after a four-element bundle lost.
The focused version, a star rating under the price plus a review carousel, moved conversion from 4.725% to 5.460% (+15.56%) across 15,615 visitors and 795 orders, with a 98% probability of being best.
Because control ran alongside it, a lucky or unlucky stretch would have hit both arms.
Paperstone's bulk discount retest works the same way.
The first attempt failed; the redesigned offer won +16.85% revenue and +18.94% AOV, with over 10,000 visitors per version.
When you see a result like this, the useful question is what changed in the variant, not whether the season changed.
A holdout group gives similar protection for changes you can't split test page by page.
OptiMonk's question-first popup for Nubest was measured against visitors who did not see it: +20.87% orders.
The sample size wasn't published, so treat the number as directional.
Regression to the mean inside A/B testing
Randomization handles regression between periods, but a related version still appears inside testing programs.
Picking the biggest winner
If you run many tests and celebrate the largest measured lifts, you are selecting results where noise added to the true effect.
When those changes ship, the realized gain tends to be smaller.
This is the winner's curse, and it is regression to the mean applied to test results.
Larger samples and pre-set stopping rules shrink it.
Stopping on an early spike
Early in a test, a few hundred visitors can produce dramatic lifts that fade as data accumulates.
Fading toward a smaller number is usually regression, not a "novelty effect" wearing off.
Deciding the sample size in advance, or using a proper sequential method, avoids stopping on the spike.
Segments that look extreme
Slice a result into twenty segments and one will look remarkable by chance.
Expect it to shrink on a retest, and treat it as a hypothesis for a new test rather than a finding.
Our page on multiple comparisons covers the correction.
How to protect your results
- Run a concurrent control. A randomized A/B test or holdout is the main defense. If you can't, compare against a similar group of pages, products or regions that didn't change over the same dates.
- Use a long baseline. Compare against several months of history, not the one bad week that prompted the change. A 12-week average is much harder to beat by luck than a single low point.
- Don't trigger changes on extremes and then measure from them. If a drop is what made you act, the drop is the worst possible baseline.
- Check the size against normal noise. Work out how much your metric moves week to week when nothing changes. An A/A test shows this directly, and our page on variance explains the calculation.
- Retest surprising winners. A real effect survives a second run. A lucky one usually shrinks.
When regression to the mean is the wrong explanation
Regression is not a reason to dismiss every improvement. It is the wrong explanation when:
- the result came from a randomized test with control and variant running at the same time;
- the baseline was a long-run average rather than a period chosen because it was unusual;
- the change is far larger than the metric's normal swings and it holds for months;
- the metric is measured on very large samples, where period-to-period noise is small.
In those cases, look at the other before/after risks instead: seasonality, promotions, traffic sources and tracking changes.
To see which results in the database come from controlled tests, browse all tests and check each entry's evidence label.