Sequential testing is a way of analyzing an A/B test that lets you check results while it runs and stop early, without raising your false positive rate above the level you chose.
It works by making the bar for significance stricter at each look (group sequential designs) or by using p-values and confidence intervals built to stay valid however often you check (always-valid inference).
The price is a slightly larger maximum sample size or wider intervals.
If you have ever opened a test dashboard on day three, seen "95% significant," and felt the urge to ship, this page is for you.
It covers the peeking problem, the two main families of sequential methods, a worked example, and when sequential testing is the wrong tool. It sits inside our A/B testing guide.
The peeking problem: why checking a normal test breaks it
A standard fixed-horizon test (a z-test or chi-square test) assumes you look at the data once, at a sample size you set in advance.
Its 5% false positive rate is only true under that assumption.
When you check repeatedly and stop the first time p drops below 0.05, you give random noise many chances to cross the line.
Armitage, McPherson and Rowe showed this formally in "Repeated Significance Tests on Accumulating Data" (Journal of the Royal Statistical Society, 1969): with no true effect, the chance of getting a significant result at some look is far higher than the nominal level.
Here is how fast it grows. We simulated 400,000 A/A tests (no real difference) and checked each one at equally spaced points, stopping at the first two-sided p < 0.05:
| Number of looks | Real false positive rate |
|---|---|
| 1 (no peeking) | 5.0% |
| 2 | 8.3% |
| 3 | 10.7% |
| 5 | 14.2% |
| 10 | 19.3% |
| 20 | 24.9% |
| 50 | 32.0% |
Checking daily over a three-week test is about 20 looks, so roughly one in four "winners" from a no-effect change would be false.
Evan Miller's well-known essay "How Not To Run an A/B Test" gives a more extreme case: if you check after every single observation, the false positive rate reaches 26.1% in the essay's example.
What peeking looks like in real tests
Our database has a clean example. Ruben ran an authority badge test (a doctor endorsement in the style of FrontrowMD) on an ecommerce product page.
After one week it showed +17% conversion rate with a 99.6% probability to beat control.
Ruben kept it running anyway. After four weeks the lift had shrunk to +1.66%, which is flat.
Called at week one, it would have been logged as a big win.
Clean Commit's shorter mobile product gallery showed +24.0% conversion rate and a 95.3% probability to beat control on 4,788 visitors, but with only 23% power, so the agency labeled it a directional signal rather than a winner.
Its ad-traffic-to-homepage test showed +103.9% conversion rate on 2,777 visitors at 7% power.
Big early numbers on small samples are exactly where peeking does the most damage, because the observed lift on a small sample swings widely (see the winner's curse).
Clean Commit
GrowthRockClean CommitClean Commit
Blend Commerce
@RubendgrfPeeking in A/B testing: what is allowed and what is not
Looking at results is not the problem. Acting on them with a test that was not designed for it is.
Three habits are safe with a fixed-horizon test:
- Checking for bugs, broken tracking, or a sample ratio mismatch.
- Stopping a test that is clearly hurting revenue, as a business call, while accepting that you learned nothing reliable.
- Watching the numbers, as long as you only make the ship decision at the planned sample size.
What breaks the test is stopping early because it looks significant, or extending it because it "almost" is.
Both change the stopping rule after seeing the data.
If you want the freedom to stop early, you need a method built for it. That is what sequential testing is.
For the planned sample size itself, use our sample size calculator and the guidance on how long to run an A/B test.
Group sequential design: planned looks with stricter thresholds
A group sequential design fixes the number of looks in advance (for example, five), and sets a stricter significance threshold at each one so the overall false positive rate stays at 5%.
The method came from clinical trials: Stuart Pocock's 1977 Biometrika paper and O'Brien and Fleming's 1979 Biometrics paper set out the two classic boundaries.
| Look (5 equally spaced) | Pocock: z needed | Pocock: nominal p | O'Brien-Fleming: z needed | O'Brien-Fleming: nominal p |
|---|---|---|---|---|
| 1 (20% of data) | 2.41 | 0.016 | 4.56 | 0.000005 |
| 2 (40%) | 2.41 | 0.016 | 3.23 | 0.0013 |
| 3 (60%) | 2.41 | 0.016 | 2.63 | 0.0085 |
| 4 (80%) | 2.41 | 0.016 | 2.28 | 0.023 |
| 5 (100%) | 2.41 | 0.016 | 2.04 | 0.041 |
Two-sided, overall alpha = 0.05. Values from our simulation; they match the standard published constants.
Pocock uses the same strict bar at every look, so it stops early more often but needs a much bigger sample if the test goes the distance.
O'Brien-Fleming is nearly impossible to pass early and close to a normal test at the end, which is why most product teams prefer it.
Lan and DeMets added the piece that made this practical online: an alpha spending function (Biometrika, 1983).
Instead of fixing the exact look times, you "spend" your 5% error budget as a function of how much of the planned sample you have collected.
That lets you look at irregular times. Spotify's experimentation team uses this approach and explains why in its comparison of sequential frameworks.
Worked example
Say your checkout converts at 3.0% and you want to detect a lift to 3.3% (10% relative), two-sided alpha 0.05, 80% power.
- Fixed-horizon test: n per arm = (1.96 + 0.84)^2 x (0.03 x 0.97 + 0.033 x 0.967) / 0.003^2, which is about 53,200 visitors per arm. One look, at the end.
- O'Brien-Fleming, 5 looks: the maximum sample is about 2.6% larger, about 54,600 per arm, checked every 10,900 per arm. If the true lift really is 10%, our simulation stopped at a mean of about 43,400 per arm, around 18% less traffic than the fixed test.
- Pocock, 5 looks: the maximum grows about 23%, to about 65,400 per arm. Expected sample at a true 10% lift is about 42,500 per arm, but if the change does nothing, you pay for the full 65,400.
At look 3 (about 32,700 per arm) with O'Brien-Fleming, you stop for a win only if z is above 2.63. A z of 2.1, which would read as "p = 0.036, significant" on a normal dashboard, means keep going.
Sequential A/B testing with always-valid p-values
Always-valid inference goes further: you can look whenever you like, as often as you like, with no planned number of looks.
Ramesh Johari, Leo Pekelis and David Walsh described it in "Always Valid Inference: Bringing Sequential Analysis to A/B Testing".
An always-valid p-value is one where the chance it ever drops below 0.05 under no effect is at most 5%, no matter when you stop.
The usual engine is the mixture sequential probability ratio test (mSPRT), a descendant of Abraham Wald's sequential probability ratio test (Annals of Mathematical Statistics, 1945).
A related approach is the confidence sequence: a confidence interval that holds at every moment of the test, not just at the end.
This is how several tools handle it, per their own documentation:
| Tool | Method | Notes from the docs |
|---|---|---|
| Optimizely | Stats Engine: sequential testing plus false discovery rate control | Built with the always-valid research above; Optimizely's launch post explains the reasoning |
| GrowthBook | Asymptotic confidence sequences (Waudby-Smith et al.) | Frequentist engine only; intervals are wider; you set a tuning parameter to the sample size where you expect to decide (default 5,000) |
| Statsig (now part of Amplitude) | mSPRT | Docs recommend it for catching regressions and time-sensitive launches, and warn early stops give uncertain effect sizes |
| Spotify Confidence | Group sequential with Lan-DeMets alpha spending | Chosen for power when the maximum sample is predictable |
The tradeoff is power. Spotify's simulations found the group sequential test was, in its words, "in most cases superior to all other methods in terms of power," while always-valid methods are the better fit when you cannot estimate the final sample size or data streams in continuously.
How to choose, and when sequential testing is the wrong choice
| Situation | Better choice |
|---|---|
| You know your weekly traffic and can commit to a max sample size | Group sequential (O'Brien-Fleming or Lan-DeMets spending) |
| Traffic is unpredictable, or you want to check at any time | Always-valid p-values or confidence sequences |
| You will genuinely only look once, at the end | Fixed-horizon test; it has the most power for the sample |
| You care mainly about catching a broken variant fast | Sequential test on a few guardrail metrics |
Sequential testing is the wrong tool, or not enough on its own, in these cases:
- You need an accurate size of the lift. Tests that stop early tend to overstate the effect, because they stop when noise happens to push the number up. Statsig's docs say the same: wait for full power if the effect size matters.
- Weekly cycles matter. A sequential method controls error statistically, but it cannot fix a sample that is all Tuesday shoppers. Ruben's badge test is the reminder: let ecommerce tests cover at least one or two full weeks even if a boundary is crossed sooner. Early effects can also be a novelty effect.
- Very low traffic. If you can barely reach a fixed sample, a design with a larger maximum makes things worse. See A/B testing with low traffic.
- Many metrics and variants. Sequential methods handle repeated looks, not multiple comparisons. You still need a correction, covered in multiple comparisons.
- You use a Bayesian tool and think it is immune. A "probability to beat control" can fool you just as Ruben's 99.6% did after one week. Read Bayesian vs frequentist A/B testing before relying on early stops.
Small-sample wins deserve a second look either way.
GrowthRock's mobile flavor link bar reached +29% orders at 98% significance in 14 days, but on only 139 vs 107 conversions, and the agency said the variant only "likely" won.
Clean Commit's featured-product swap showed +5.6% conversion after 11 days and 38,732 visitors at a 71.1% probability, which the agency recorded as an early signal, not a result.
For the full list of ways a test can mislead you, see common A/B testing mistakes, and browse all tests in the database to see how practitioners report significance and duration.