Test and learn means treating every change as a bet you check with a controlled experiment, then keeping the lesson whether the bet wins, loses or comes out flat.
An experimentation program is the system that makes this routine: a shared way to pick ideas, run trustworthy tests, decide, and record what was learned.
Most tests do not win, so the program's real output is decisions and knowledge, not a string of uplifts.
This guide covers the culture behind it, how to set the program up, how fast to test, how to measure velocity without fooling yourself, and where your team sits on a maturity model.
Every point is backed by real results from our test database.
Experimentation culture: why most ideas need a test
The case for a test-and-learn culture starts with one uncomfortable number: most ideas don't work.
A KDD 2022 paper by Ron Kohavi, Alex Deng and Lukas Vermeer collects published success rates: about one third of experiments at Microsoft improved the metric they targeted, about 15% at Bing, about 8% in Airbnb search, and roughly 10% at Booking.com, Google Ads and Netflix.
Optimizely's analysis of 127,000+ experiments run between 2018 and 2023 found that about 12% win on their primary metric.
Spotify's experimentation team reports a similar win rate of about 12%, but also that about 64% of experiments produce a clear learning.
That gap is the whole argument for culture: if you only value winners, nearly nine in ten tests look like waste.
Our own database shows why you can't judge this from what people post.
Of 1,724 results, 1,447 are marked as wins, because practitioners mostly share their winners.
Clean Commit is an exception: the agency publishes its whole library, and of their 173 tests in our database, 16 won, 35 lost and 122 were flat.
That is about a 9% win rate, right in line with the big-company figures.
Those losses are often the most useful results.
Clean Commit replaced a trust-focused homepage hero with a sales carousel and saw conversion fall 46.2% (on a small sample of 1,082 visitors, so the size of the drop is uncertain).
A cleaner single-row header with a mega-menu cut mobile conversion by 22.5%.
Both look like reasonable design upgrades. Without a test, both would have shipped.
A healthy culture has a few visible habits:
- Losses are reported, not buried. Dave Diederen tested removing an email popup because store owners assumed shoppers hated it. It lost 20.51% revenue per visitor, so the team ran it again on a second store, and it lost again.
- Opinions lose to data, including senior ones. Harvard Business School professor Stefan Thomke's HBR article "Building a Culture of Experimentation" uses Booking.com, which runs about 25,000 tests a year, as its lead example of a company where testing is part of everyday product work.
- A flat result is a decision. Flat means "this change doesn't matter enough to detect," which frees you to ship the cheaper option or move on.
The experimentation program: roles, process and rules
Culture is the attitude. The program is the machinery that turns it into weekly work. At minimum, it needs five parts.
| Part | What it does | Common failure |
|---|---|---|
| Idea intake and prioritization | Collects ideas from research and ranks them with a framework such as ICE | The loudest person's idea goes first |
| A written hypothesis per test | States the change, the expected effect and the reason | Tests with no reason can't teach you anything when they lose |
| Metrics, set before launch | One primary metric plus guardrail metrics | Picking whichever metric moved after the fact |
| Decision rules | Sample size, run time and what counts as a win, fixed in advance | Peeking and stopping on the first good day |
| A learning repository | Stores every result, including losses and flats, with the reasons | Re-running the same failed idea a year later |
The primary metric matters more than most teams expect.
Patrick McKenzie's 2008 redesign of Bingo Card Creator lifted trial downloads from about 100 to 125-150 a day, while sales dropped from about 2 a day to slightly under 1.
That was a before-and-after comparison of a bundled change, not a clean A/B test, but it shows the trap: a team watching downloads would have called it a win.
The same program later ran a cleaner test that paid off for years.
Cutting the free trial from 15 cards to 8 raised trial-to-paid conversion from 1.04% to 1.89%, at 99% confidence.
Across the 23 Bingo Card Creator results in our database, 11 won, 7 lost and 5 were flat, a realistic mix for a one-person program.
The repository is what makes test and learn compound.
Dave Diederen's countdown timer results show why. In the US, a checkout timer raised revenue per visitor 3.16%, and an "order reserved" version raised it 4.83%.
Another checkout timer test lost 7.1%, and across 20+ brands in a European market a timer lost 35.61%.
One result alone would mislead you. The pattern across the log tells you where the idea works.
@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen
Clean CommitIf you need a step-by-step version of the testing loop itself, see the CRO process.
Testing velocity: how many tests you can run
Testing velocity is the number of experiments you launch (or complete) in a period, usually counted per month or per year.
Optimizely's benchmark report gives a sense of the spread: the median company runs 34 experiments a year, about 3 a month.
Reaching the top 10% takes around 200 tests a year (16+ a month), and the top 3% run over 500.
Treat those as vendor benchmarks from Optimizely's own customers, not a target.
Your real ceiling is traffic. Each test needs enough visitors to detect the effect you care about, and you can't reuse the same visitors on the same page for two conflicting tests.
A quick rule of thumb for sample size at 95% confidence and 80% power, cited in the Kohavi, Deng and Vermeer paper, is:
n per variant = 16 x p(1 - p) / delta^2
Here p is your baseline conversion rate and delta is the absolute change you want to detect. A worked example:
- Product page traffic: 200,000 visitors a month.
- Baseline conversion rate: 3% (p = 0.03), so p(1 - p) = 0.0291.
- Smallest lift worth detecting: 10% relative, which is 0.3 percentage points (delta = 0.003).
- n per variant = 16 x 0.0291 / 0.000009 = about 51,700. An A/B test needs about 103,500 visitors.
- That page can support about 2 tests a month, or roughly 23 a year.
To go faster, you have four honest levers: test on more pages or audiences at once, test bolder changes with a larger minimum detectable effect, cut the time between tests (setup, QA, analysis), and stop running tests you could decide without one.
Lowering your evidence bar is not a lever.
If traffic is the hard limit, read the low-traffic testing guide first.
This rule of thumb is the wrong tool when your metric isn't a simple conversion rate.
For revenue per visitor, the variance is much higher than p(1 - p), so you need a proper power calculation from your own data.
Experiment velocity: measuring speed without gaming it
Velocity is easy to count and easy to inflate.
Split one test into three button-color tests and your velocity triples while your learning doesn't.
So track it alongside metrics that capture quality.
| Metric | How to calculate | What it tells you |
|---|---|---|
| Velocity | Tests launched per month | Throughput |
| Win rate | Wins on the primary metric / completed tests | Idea quality (expect roughly 10-15%) |
| Learning rate | Tests with a clear decision and a recorded lesson / completed tests | Whether flats and losses still pay off |
| Cycle time | Days from idea approved to decision | Where the process is slow |
| Valid test rate | Tests without setup problems such as sample ratio mismatch / all tests | Whether results can be trusted |
Optimizely's data also suggests velocity alone isn't where the value is.
In that report, 77% of experiments tested only two variations, yet tests with four or more variations were about twice as likely to win, and the top 5% by uplift tended to involve larger code changes.
Fewer, bolder tests can beat many small ones.
Be careful with win rate as a team goal.
When wins are rare, many significant results are false positives: the Kohavi paper estimates that at an 8% success rate, about 26% of significant wins at p < 0.05 are false.
A team rewarded for wins will drift toward peeking and cherry-picking.
Reward valid, well-documented decisions instead, and watch for the winner's curse when you forecast impact.
CRO maturity model: crawl, walk, run, fly
The best-known experimentation maturity model comes from Aleksander Fabijan and colleagues, based on Microsoft's experimentation team.
Their 2017 ICSE paper, "The Evolution of Continuous Experimentation in Software Product Development", describes four phases across technical, organizational and business dimensions.
The authors note it was inspired by an internal model created by Ronny Kohavi.
| Phase | What it looks like | Who runs tests | Metrics |
|---|---|---|---|
| Crawl | First tests, often without a platform; the goal is to prove value and earn funding. Fix tracking first. | A central data or CRO specialist | A few key signals |
| Walk | A basic platform with power analysis and pre-test A/A checks | Product managers create tests; specialists still run and analyze them | Success, guardrail and data quality metrics |
| Run | Experimentation becomes the norm for new features; the paper says scaling past 100 experiments a year needs alerting and better tooling | Product teams run their own tests; analysis still reviewed by specialists | An overall evaluation criterion tied to long-term goals |
| Fly | Nearly every change is tested; automatic shutdown of harmful tests and institutional memory are built in | Teams across the company, self-serve | A stable criterion, changed on a fixed schedule |
The paper's own example of "fly" is Bing, where teams ran over 10,000 experiments a year.
Most ecommerce stores and SaaS companies will never need that.
For them, the model is most useful as a checklist of what to fix next.
A practical way to place yourself:
- Crawl: you run occasional tests, tracking is patchy, and nobody owns the program. Fix tracking, run an A/A test, and agree on one primary metric.
- Walk: you test monthly with a written hypothesis and fixed decision rules. Start the learning log and a prioritized backlog.
- Run: several teams test, results are shared, and losses are discussed openly. Add guardrails, QA checks and iterative follow-up tests.
- Fly: major launches don't ship without a test and the repository shapes the roadmap. Protect quality as volume grows.
Many teams at the crawl or walk stage bring in outside help to set the process up.
If that's you, our CRO consultants directory lists practitioners whose tests appear in this database.
For a broader view of methods and statistics, start at the A/B testing hub.