Iterative testing is running a series of A/B tests on the same hypothesis, where each test is designed from what the last one taught you.
Instead of moving to an unrelated idea after every result, you push a winner further, rework a loser, or retest a result you don't trust yet.
The unit of work is the question you're answering, not the single test.
This page shows how that works with real follow-up tests from our database of A/B test results: stores, SaaS teams and app developers who lost, changed one thing, and won on the next round, or won and kept going.
Iterative A/B testing: one hypothesis, several tests
A single test answers a narrow question: did this exact version beat that exact control, on this traffic, in these weeks?
It rarely tells you whether the idea behind it was right.
A losing variant can be a good idea with a bad execution, and a winning one can be a lucky draw.
That matters because most ideas don't work the first time.
In a Microsoft paper on its experimentation platform, Ronny Kohavi and colleagues wrote that "only about 1/3 of ideas improve the metrics they were designed to improve", with about a third flat and a third negative.
Optimizely's Evolution of Experimentation report, built on its customers' experiment data, puts the share of experiments that win on the primary metric at around 12%.
If most first attempts fail, dropping an idea after one test throws away most of what you paid to learn.
Iteration is how you get value from the other two-thirds.
Iterative testing vs. random testing
The opposite of iterative testing is sometimes called "spaghetti testing": throwing unrelated ideas at the site to see what sticks.
Each test starts from zero, so a loss teaches nothing about the next test.
| Random (one-off) testing | Iterative testing | |
|---|---|---|
| What drives the next test | Whatever idea is next on the list | What the last test showed about the same problem |
| After a loss | Idea is dropped | Diagnose why, change the execution, test again |
| After a win | Ship and move on | Ship, then push the same lever further |
| What you build up | A list of unrelated results | Knowledge about what your customers respond to |
Iteration starts with a clear hypothesis. If you can't say which customer problem a test was meant to solve, you can't say what the next version should change.
The three loops
Every result points to a different next step:
- After a win: exploit it. Push the same idea further, extend it to other pages, or combine it with other winners.
- After a loss: diagnose it. Was the idea wrong, or the execution? Change the part you think failed and test again.
- After a flat result: decide whether the change was too small to matter, the test too small to detect it, or the idea simply irrelevant to buyers.
The examples below follow each loop with tests from our database.
Conversionista
Conversion Fanatics
ConvertiblesTest iteration: how to plan the next round
After a loss: change the execution, keep the hypothesis
The clearest case for iteration is a losing test where the idea survives and only the delivery changes.
Paperstone first offered bulk-buy discounts as a yellow ribbon in the right sidebar.
It made no difference to conversions, and the team put that down to banner blindness.
They kept the offer, redesigned how it was shown, and the retest lifted revenue 16.85% and average order value 18.94%, with over 10,000 visitors per version.
Conversion Fanatics had a similar path on a mobile product page where shoppers had to tap to open each option, such as size and color.
Expanding every option group lost by about 4%, after leading by more than 40% early on.
Expanding only the key option groups, with easier-to-tap choices, then produced +36.6% mobile conversions and +6.4% mobile revenue per visitor.
The hypothesis (less tapping helps) held up; showing everything was the problem.
Conversionista's test for the energy company Fortum shows how test data can point straight to the fix.
A redesigned pricing page that moved price details into a modal cut click-through 9%.
Of the people who opened the deeper price details, 18% went on in the variant against 75% in the original.
The next version kept the clearer price blocks but brought back the drop-down for details, and click-through rose 9%.
Pricing tests often iterate on size rather than design.
Levain Bakery, testing through ABConvert, saw revenue per visitor fall with a large price increase.
A smaller increase raised revenue per visitor slightly, and the new price was adopted.
The source doesn't give the sizes or the lift, but the sequence is the point: the first test set an upper bound, the second found a price that worked. More on that in price testing.
After a bundled loss: split it up
When a variant changes four things and loses, you don't know which one hurt.
Convertibles hit this on a whisky subscription product page: a four-element bundle lost, so they reran just the review elements, a clickable "4.9 from 300+ reviews" row under the price plus a member review carousel below the button.
That version moved conversion from 4.725% to 5.460% (+15.56%) across 15,615 visitors and 795 orders, with a 98% probability of being best.
Splitting a bundle trades speed for clarity. Use it when a bundle loses and you need to know which part to keep.
After a win: push the same lever further
A win tells you a lever works. The next tests ask how far it goes.
Conversio's work for Hunter Boots added popularity-based social proof messages to product pages and saw +9% conversion in the UK and +6% in the US.
A follow-up rotated two different social proof metrics and added a further 2% in the UK and 1% in the US.
Note the shape: the second step was much smaller than the first, which is normal as you approach the limit of a lever.
Jagex did the same on RuneScape's Treasure Hunter page.
Showing four key packages on the page instead of only on the payment step gave about +10% purchases.
The winner became the new control, and marking one package as Recommended added about 6% more.
POSist followed a winning homepage test (a shorter page with a sharper hero, +16% visits to the demo page) with a second round that added logos, testimonials and split paths, worth about +5% Contact Us visits in a one-week test.
A week is short, so treat that second number with care.
Stack winners into a new control
App teams often iterate fastest because a paywall is one screen with one clear metric.
Stardust's paywall tests show a full chain. Adding a 7-day free trial lifted paywall conversion 131%.
A discount badge with weekly-equivalent pricing gave +50% conversion and +62.5% proceeds per user, while a yearly-only paywall lost.
Combining the winners with a video background then produced +67% paywall conversion and +12.5% proceeds per user; the combined winners without video gave +55.7% and +9%.
A trial timeline design tested against that new control added another 13.6% conversion, with trial-to-paid unchanged.
Each step tested against the current best version, not the original page.
That is what makes the gains build on each other instead of overlapping.
It also keeps you honest: if a combination underperforms its parts, you find out.
After a surprising win: retest before you build on it
Iteration builds on earlier results, so a false positive at step one sends every later test the wrong way.
The risk is larger than most teams assume.
In "A/B Testing Intuition Busters", Kohavi, Deng and Vermeer estimate the false positive risk of a statistically significant result: with a 10% success rate (their figure for Booking.com, Google Ads and Netflix), it is 22.0% at 80% power and 52.9% at 20% power.
See statistical power for why underpowered tests make this worse.
Voices.com did this well. Removing a row of featured voice actors from the homepage gave +33% sign-ups in a 10-day, 50/50 test, and they reran it to validate: +28%.
The second number is smaller, which is what you should expect.
Big first results usually shrink, a pattern covered in the winner's curse.
Retests can also cut the other way. Apollo Education Group ran a birthday email with the subscriber's name in the preheader; the first run did poorly, and a retest months later won with +9% opens.
When two runs of the same test disagree, the honest reading is that the true effect is small and uncertain, not that the second run is the real one.
When to stop iterating
Iteration has a failure mode: polishing one small idea long after it has stopped paying.
Optimizely's report says low-hanging fruit dries up, that "you can only change the color or text on a button so many times until you are at a local maximum", and that experiments with bigger, more complex changes and more variations tend to produce larger uplifts.
Tests with four or more variations were twice as likely to win on average, yet over 77% of experiments tested only two.
So iterate on the hypothesis, not on the pixel. Two practical rules:
- Test several executions at once when traffic allows. An A/B/n test with three versions of the same idea answers in one round what three sequential A/B tests would.
- Switch to a bigger change when follow-ups keep shrinking or keep failing. VacationRoost's first four attempts at its lead form, including a shorter form and a borrowed layout, failed to beat the original. The fifth rebuilt the whole page around a conversion framework and won +19%.
Stop iterating on a hypothesis when two or three well-powered tests with different executions all come back flat.
At that point the most likely answer is that the problem you were solving isn't the one holding buyers back.
Keep the chain in one place
Iteration only works if the next person can see what came before: the hypothesis, every version tested, the numbers, and why the team thinks each one won or lost.
A learning repository holds that history, and a testing roadmap reserves slots for follow-ups so they don't get pushed out by new ideas.
For the bigger picture of running tests as a program, see the A/B testing guide.