A holdout group is a small, randomly chosen slice of users who are kept on the old experience while everyone else gets the changes you test and ship.
Comparing the two groups after weeks or months shows the real, combined effect of those changes, which is usually smaller than the sum of the individual test wins.
It is a control group for your whole test program, not for one test.
What holdout testing is
In a normal A/B test, the control and variant split traffic for a few weeks, you pick a winner, and the control disappears.
A holdout keeps a control alive after the decision.
Those users keep seeing the old version while every winner rolls out to everyone else.
Statsig's documentation describes it as withholding "a small percentage of users, typically 1% to 2%, from a set of new features so you can measure the features' combined effect over months."
The held-out users are the control. An equal-sized sample of users who received the features is the test group.
Holdout testing answers questions a single test can't:
- Did the five winners we shipped this quarter add up to the lift we reported?
- Does a lift that showed up in a two-week test still exist after three months?
- Is a tool, a popup or a channel earning more than it costs, compared with not having it at all?
Why summed test wins overstate the total
Teams that add up the lifts of every shipped winner almost always overstate their impact.
Part of the reason is selection: you only ship tests that looked good, and a result that looked good often got lucky.
Minyong R. Lee and Milan Shen of Airbnb studied this in "Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments" (KDD 2018).
The paper looks at the combined impact of launched features, shows that picking significant winners biases that estimate upward, and proposes a correction used in Airbnb's experiment reporting platform.
A holdout measures the total directly, so it doesn't depend on a correction model.
It also picks up things a short test misses, like a novelty effect that fades, or two winners that cancel each other out.
Our page on the winner's curse covers the selection problem in more depth.
What a holdout looks like in our database
Most tests in the LiftArchive database compare two versions of a page.
A few compare a change against nothing at all, which is the same logic a holdout uses.
- NuBest's question-first discount popup, published by OptiMonk, was measured against visitors who never saw the popup. Orders were +20.87% higher in the popup group. The source doesn't give a sample size, so treat it as directional.
- Adobe's segmented Creative Suite 3 launch emails were compared with a no-contact control group that got no launch email. The emailed group had a +41% overall order rate and +72% direct orders. That shows the campaign was worth sending. It doesn't show the segmentation beat a generic email, because no generic email was tested.
- Kasper Dolk's "upsells vs no upsells" test ran about 398.6k visitors per arm. Revenue per visitor moved from $1.46 to $1.49 (+1.86%) and conversion rate from 1.50% to 1.52%. This is how you check whether an upsell app pays for itself.
The Adobe case shows the main thing to remember.
A holdout tells you whether the whole package beats doing nothing.
It doesn't tell you which part of the package did the work.
@kasper_dolk@kasper_dolk
OptiMonkGlobal holdouts vs local holdouts
A global holdout excludes a group of users from every experiment and feature rollout.
A local (or selected) holdout excludes them only from the specific features or tests you pick.
| Global holdout | Local / selected holdout | |
|---|---|---|
| Who is held back | From every new test and rollout | From a chosen set of features |
| Answers | What is our whole program worth? | What is this initiative worth? |
| Typical use | Annual or quarterly program ROI | A personalization engine, a redesign track, a new tool |
| Main cost | Held-out users miss every winner | Held-out users miss only the chosen features |
Vendors implement this in different ways. Optimizely Feature Experimentation's global holdouts set aside users who "do not experience any experiments or new feature rollouts," with a warning if you go above about 5% of traffic.
A holdout can't be paused, only concluded, and a concluded holdout can't be turned back on.
Optimizely also offers local holdouts that apply only to the rules you choose.
In Statsig, a global holdout "captures the aggregate impact of all features developed after the holdout began" and is enforced on new gates, experiments and layers with the same unit type.
A selected holdout covers only the features you list.
Statsig lets you set the holdout between 1% and 50% and recommends running it for three to six months, then releasing it.
(Statsig is now owned by Amplitude, so check the current docs if you use it.)
If your testing tool has no holdout feature, you can build a basic one with feature flags.
Bucket a fixed percentage of user IDs into a "holdout" flag, and exclude that flag from every test audience and rollout.
Holdout group vs control group
A control group is the untreated comparison group in any experiment.
In an A/B test it is the version without the change, and it lasts as long as the test does.
A holdout group is a control group that outlives the test.
It stays on the old experience after the winner ships, so you can keep measuring.
| Control group (A/B test) | Holdout group | |
|---|---|---|
| Compares against | One variant | Everything shipped since the holdout began |
| Size | Often 50% of test traffic | Usually 1-10% of all users |
| Duration | Days to weeks | Months |
| Question | Is B better than A? | Did all our changes together help, and does it last? |
A "holdback" is a related term. It usually means a small group kept off one specific feature after launch, a local holdout of one. Some teams use the two words interchangeably.
Sizing a holdout: a worked example
Holdouts are small, so they have low statistical power.
Check what lift you can actually detect before you start.
For conversion rates, the standard error of the difference between two groups is:
SE = sqrt( p × (1 - p) × (1/n_holdout + 1/n_rest) )
At 95% confidence (two-sided) and 80% power, the smallest detectable absolute difference is roughly 2.8 × SE (1.96 + 0.84).
Say a store gets 200,000 unique visitors a month at a 3% conversion rate and holds out 5%:
| Holdout share | Duration | Holdout users | Rest | Detectable change | Relative |
|---|---|---|---|---|---|
| 5% | 1 month | 10,000 | 190,000 | 0.49 points | 16.3% |
| 5% | 3 months | 30,000 | 570,000 | 0.28 points | 9.4% |
| 5% | 6 months | 60,000 | 1,140,000 | 0.20 points | 6.7% |
| 10% | 3 months | 60,000 | 540,000 | 0.21 points | 6.9% |
This simple version assumes each visitor is new, and real returning traffic will give you fewer unique users.
The takeaway still holds. A 5% holdout on a mid-size store can only detect a combined lift of around 9% after three months.
If your shipped winners add up to less than that, the holdout will probably come back inconclusive.
You can run the same numbers with our sample size calculator, entering the holdout as one arm.
Holdouts also cost money. Held-out users miss every winner.
In Kasper Dolk's upsell test, the upsell version was estimated at +$17,775 a month in profit.
A 5% holdout on that change would give up roughly 5% of it, about $890 a month, in return for a cleaner long-run measurement.
When a holdout is the wrong choice
- Low traffic. If a 5-10% slice won't reach a useful detectable effect within six months, the holdout will cost revenue and teach you nothing. Fix your test velocity first. See testing with low traffic.
- Legal, security or pricing-parity changes. You can't keep 5% of users on a broken checkout, an outdated privacy flow or a price you've stopped honoring.
- Features users already rely on. Statsig's docs warn against holding out features that have already shipped, because that "turns off features that users already use." Start the holdout before the features go live.
- Short sales cycles with no repeat visits. If your users visit once, a long holdout adds little over the original test.
- You need to know which change worked. A holdout measures the bundle. Use regular A/B tests to judge individual changes.
Running a holdout well
- Randomize by a stable user ID, not by session or cookie alone. Otherwise users drift between groups and the holdout gets contaminated.
- Set it up before the test cycle starts, and agree in advance on the duration (three to six months is a common range) and the primary metric, such as revenue per visitor or retention.
- Exclude the holdout from every test audience. If held-out users land in experiments, they're no longer a clean baseline.
- Check the split. If the holdout should be 5% and comes in at 4.6% of users, look for a sample ratio mismatch before trusting any result.
- Watch guardrail metrics like refunds, unsubscribes and support contacts. A program can lift conversion while slowly hurting retention, and a holdout is one of the few ways to see that.
- Report the interval, not just the point estimate. "+4%, 95% interval -1% to +9%" is an honest program report. "+4%" alone isn't.
Some vendor results in our database have no concurrent control at all.
Bricomarché's AI recommendations rollout, published as a case study, reported +21% homepage conversion as a before-and-after comparison.
Seasonality and other site changes are mixed into that number.
Running a small holdout during a rollout like that one gives you a number you can defend.
Holdouts belong in a mature experimentation program. They're how you show leadership the testing budget pays off, using evidence rather than a spreadsheet of summed lifts.
For more on running individual tests, start at our A/B testing guide.