CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction method for A/B tests.
It subtracts the part of each user's result that you could already predict from their behavior before the test, so the same experiment reaches the same statistical power with fewer users or in less time.
Microsoft researchers introduced it in 2013 and reported about 50% lower variance on Bing, which is like getting the same power from half the traffic.
It does not change what your test measures.
It removes noise that has nothing to do with the treatment, and how much it helps depends on how well past behavior predicts behavior during the test.
How CUPED variance reduction works
Every A/B test metric mixes two things: the effect of your change and the natural differences between users.
A heavy buyer spends more than a light buyer in either variant, and that spread is what makes small lifts hard to detect.
You can read more about this spread on our variance page.
CUPED uses a covariate, usually the same metric measured for the same user before the test started.
If someone spent a lot last month, part of what they spend this month is predictable.
CUPED takes out that predictable part and compares only what is left.
The method comes from the paper "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data" by Alex Deng, Ya Xu, Ron Kohavi and Toby Walker, presented at WSDM 2013. It adapts the control variates technique from Monte Carlo simulation to online experiments.
The formula in plain text
For each user, with Y = the metric during the test and X = the same metric before the test:
- Adjusted metric: Y_cuped = Y − θ × (X − mean(X))
- Best θ: θ = cov(X, Y) / var(X), estimated on pooled data from all variants
- Variance after adjustment: var(Y_cuped) = var(Y) × (1 − ρ²), where ρ is the correlation between X and Y
Then you run your usual test (t-test, sequential test or Bayesian model) on Y_cuped instead of Y. Because X was measured before randomization, its average is the same in both variants in expectation, so the adjustment does not bias the difference between them.
The paper stresses that the same θ must be used for control and treatment.
The θ above is the slope of an ordinary least-squares regression of Y on X. That is why CUPED is often described as regression adjustment, or a special case of ANCOVA.
What a given correlation buys you
Required sample size scales with variance, so the (1 − ρ²) factor directly tells you how much traffic you save.
The minimum detectable effect scales with the standard deviation, so it shrinks by the square root of that factor.
| Correlation ρ (pre vs. during) | Variance left (1 − ρ²) | Traffic saved for same power | MDE multiplier |
|---|---|---|---|
| 0.2 | 0.96 | 4% | 0.98 |
| 0.4 | 0.84 | 16% | 0.92 |
| 0.6 | 0.64 | 36% | 0.80 |
| 0.7 | 0.51 | 49% | 0.71 |
| 0.8 | 0.36 | 64% | 0.60 |
Bing's roughly 50% reduction corresponds to a correlation of about 0.7. A correlation of 0.2 is close to useless, which matters for the ecommerce caveat below.
A worked example
Say your store tests a new product page on revenue per visitor.
Spend per visitor during the test has a standard deviation of $20, spend in the 14 days before has the same spread, and the correlation between the two is 0.6.
- cov(X, Y) = 0.6 × 20 × 20 = 240, and var(X) = 400, so θ = 240 / 400 = 0.6.
- Average pre-period spend is $30. A visitor who spent $50 before the test and $45 during it gets an adjusted value of 45 − 0.6 × (50 − 30) = $33.
- A visitor who spent $10 before and $15 during gets 15 − 0.6 × (10 − 30) = $27.
- Variance drops from 400 to 400 × 0.64 = 256, so the standard deviation falls from $20 to $16.
If your sample size calculation called for 20,000 visitors per variant, CUPED at this correlation needs about 12,800 for the same power.
Or, keeping 20,000 visitors, the smallest lift you can reliably detect falls to 80% of what it was (a 5% MDE becomes 4%).
CUPED in A/B testing: where it helps and where it doesn't
CUPED is most useful when the lift you care about is small relative to the noise.
Our database has plenty of tests that landed in that zone and ended up inconclusive:
- Blend Commerce showed trending searches and product suggestions as soon as the search bar opened for Fragrance Hub. The result was +1.92% revenue per visitor, not statistically significant after about two weeks.
- Convertibles added a "Help me choose" comparison modal for a sports equipment brand: +2.6% revenue per visitor, about 530 orders per arm, 62% probability to beat control, counted as flat.
- Dave Diederen placed a review rating above the product title: no conversion rate change, +1.30% revenue per visitor.
Lifts of 1-3% on revenue are exactly the size where extra power decides whether you learn anything.
But we can't say CUPED would have made any of these significant.
That depends on how much pre-test history those stores had for the visitors in each test, and the tests don't report it.
Compare Dave Diederen's checkout countdown timer, which won at +3.16% revenue per visitor and +2.99% conversion rate across repeated runs.
Repeating a test is another way to buy confidence in a small effect.
CUPED tries to get there with less traffic.
@DaveDiederen
Blend Commerce@DaveDiederenBlend Commerce
ConvertiblesThe ecommerce catch: new visitors and revenue
CUPED only reduces variance for users who appear in the pre-period.
The original paper names experiments on new users and retention metrics as cases with no pre-experiment data to work with.
Many store tests are dominated by first-time visitors from paid traffic, and those visitors get no adjustment.
Revenue is also a weak case even when history exists.
In the Bing paper, CUPED cut variance on revenue-per-user by less than 5%, because revenue before and during the test was poorly correlated.
Purchases are rare and lumpy, so last month's spend says little about this month's.
Segments with history are where CUPED has the most to work with.
Blend Commerce's PerTronix test, which exposed the mobile search bar, reported +45% AOV and +46% revenue for returning visitors.
Only the returning-visitor figures were published. Returning visitors are the group a CUPED adjustment could actually tighten.
When CUPED is the wrong choice
- Mostly new users. No history means no adjustment. Low-traffic stores with mostly new visitors gain little; see A/B testing with low traffic for other options.
- Covariates the treatment can affect. The paper warns never to use these: it shows an example where doing so flipped the direction of the conclusion. Anything measured after a user is exposed to the test is off limits, unless the treatment can't change it, like the day they entered the test.
- Weakly correlated metrics. If ρ is under about 0.3, the savings are a few percent. Don't plan test durations around CUPED until you have measured the correlation on your own data.
- Interpreting adjusted means as real values. The adjusted metric is for comparing variants. Report the lift, not "adjusted revenue per visitor" as if it were what customers spent.
CUPED also doesn't fix the usual problems. Peeking, sample ratio mismatch and stopping early still break a test with CUPED on.
If you check results often, pair it with a proper sequential test.
Variance reduction in A/B testing beyond CUPED
CUPED is one member of a family of methods that buy statistical power without more traffic:
- Stratification and post-stratification. Split users into groups (country, device, new vs. returning) and combine within-group estimates. The Bing paper handles missing pre-period data this way: an indicator for "had pre-period data" is the same as splitting users into two strata.
- Regression with several covariates. Extends CUPED with more predictors measured before exposure. The paper found the same metric from the pre-period was the best single covariate, and adding entry-day on top gained only 2-3 percentage points of extra reduction.
- Winsorizing or capping outliers. Capping extreme order values cuts variance on revenue metrics, often more than CUPED does for stores. It changes the metric slightly, so set the cap before the test.
- Choosing a less noisy metric. Conversion rate or add-to-cart rate usually has less relative noise than revenue per visitor, at the cost of measuring something closer to the change than to revenue.
An A/A test is a good way to check any of these on your own data: run it with and without the adjustment and compare the confidence interval widths.
How testing tools implement it
Tools differ in pre-period length, which metrics qualify, and how they treat users without history.
As of November 2026, according to their documentation:
| Tool | Default pre-period | Notes from the docs |
|---|---|---|
| Statsig | 7 days before each user's exposure | Applied automatically to key metrics; uses stratification for users without pre-experiment data; window customizable on Warehouse Native |
| GrowthBook | 14 days of pre-exposure data | Pro and Enterprise plans; post-stratification on Enterprise; notes CUPED is less powerful for rare metrics and new users |
| Optimizely | Two weeks before experiment start | Numeric metrics only, not conversion metrics; applies to the first three decision metrics by default; no covariate-imbalance health check |
| PostHog | 14 days (configurable 1-30) | Mean and funnel metrics; ratio and retention metrics are left unadjusted |
The pre-period choice matters. The Bing paper found a longer pre-period raised the share of users with history, and recommended 1-2 weeks: too short gives poor matching, too long weakens the correlation with the test period.
Checklist before you switch it on
- Measure what share of your test users had activity in the pre-period. If it's small, expect little benefit.
- Compute the correlation between pre-period and test-period values for your primary metric on past data.
- Use only covariates measured before exposure, and the same θ for every variant.
- Plan test duration on the unadjusted variance until you've seen the reduction in practice.
- Report the adjusted lift and its interval, plus the unadjusted result, so readers can see what the adjustment did.
For more on the statistics behind your tests, start at our A/B testing guide, or browse all test results to see the effect sizes real stores get.