A testing roadmap is a dated plan of which A/B tests you will run, on which pages, in what order, for the next quarter.
It is sized to your traffic, so every slot can actually reach a result, and it leaves room for follow-up tests, because most first attempts lose or come back flat.
This guide shows how to build one for a 13-week quarter, using real sequences from our database where a losing test set up the winner that came after it.
What an experimentation roadmap is (and what it is not)
An experimentation roadmap sits between your idea backlog and your test calendar.
The backlog holds every idea. The roadmap holds the handful you have committed to, with a page, a start week, a primary metric and an owner.
It is not a list of design changes you have already decided to ship.
If the team would launch the change whatever the result, it does not belong on the roadmap.
Leave it out and save the traffic for questions that are still open.
A useful roadmap answers four questions for each slot:
- What are we testing? A written hypothesis, not just "new hero".
- Where? One page or flow, so tests on the same page don't overlap.
- How long? A run time worked out from traffic and the smallest lift worth finding.
- What happens next? What the team does if it wins, loses or comes back flat.
The roadmap is one part of a wider experimentation program.
Microsoft researchers describe how programs mature in four stages they call crawl, walk, run and fly, with Bing running more than 10,000 experiments a year at the top end (Fabijan et al., ICSE 2017).
Most teams reading this are at crawl or walk, and a quarterly roadmap is the tool that gets them to run.
How many tests should a quarter hold?
Less than most teams think. Optimizely's analysis of experiments run on its platform found the median company runs 34 experiments a year, about three a month.
To be in the top 10% for velocity, a company needs around 200 a year (Optimizely, Evolution of Experimentation report).
Three a month is roughly nine or ten a quarter.
That is a realistic target for a team with one analyst and shared developer time.
If your traffic is low, the honest number may be four.
Building an A/B testing roadmap for one quarter
Build the quarter in this order: traffic first, then ideas, then the calendar.
Teams that start with ideas end up with 30 tests and room for six.
Step 1: Work out how many test slots each page has
Every page can only hold so many tests in 13 weeks.
To size them, use a standard sample-size rule.
Lehr's rule of thumb, from Gerald van Belle's Statistical Rules of Thumb (chapter 2), gives the sample per group for 80% power at a two-sided 5% significance level:
n per variant ≈ 16 × p × (1 − p) ÷ d²
Here p is the baseline conversion rate and d is the absolute difference you want to detect.
Use a proper calculator before launch; this rule is for planning.
Worked example (our calculation): a product page gets 15,000 visitors a week and converts at 3%.
| Smallest lift worth finding | d (absolute) | Visitors per variant | Total, A vs B | Weeks per test | Tests per quarter on this page |
|---|---|---|---|---|---|
| 10% relative (3.0% to 3.3%) | 0.003 | about 51,700 | about 103,500 | 7 | 1 |
| 20% relative (3.0% to 3.6%) | 0.006 | about 12,900 | about 25,900 | 2 | 6 |
Working: 16 × 0.03 × 0.97 = 0.4656. Divided by 0.003² (0.000009) that is about 51,700 per variant.
Halving d multiplies the sample by four. Run times are rounded up to whole weeks so each test covers full weekly cycles.
The table is the whole argument for planning by traffic.
The same page holds one subtle test or six bold ones.
Choosing your minimum detectable effect per page is the biggest decision on the roadmap.
When this rule is the wrong choice: it assumes a 50/50 split, two variants and one primary metric.
For revenue per visitor, which has much higher variance than a conversion rate, it understates the sample you need.
For tests with three or more variants, or if you plan to check results early, use a calculator built for that design.
Step 2: Fill the slots from a ranked backlog
Score the backlog with a simple prioritization method, then place the top ideas into the slots you sized.
Bold ideas go on pages where you can only afford a large MDE.
Small copy and layout changes go on your highest-traffic pages, the only places they can be measured.
Keep one test per page at a time unless your tool supports mutually exclusive experiments.
Spread tests across the funnel (landing, product, cart, checkout) so the program isn't stuck waiting on one page.
Step 3: Reserve slots for follow-up tests
This is the step most roadmaps skip. Optimizely reports that around 12% of experiments win on the primary metric (same report).
Clean Commit, which publishes its whole test library, shows a similar picture: of its 173 tests in our database, 16 won, 35 lost and 122 were flat.
See our A/B test win rate research for more.
So plan for it. Leave roughly one slot in four empty at the start of the quarter, and fill it with the follow-up to whichever test taught you the most.
Our database has several cases where the second attempt worked:
- Conversionista moved Fortum's price details into a modal and click-through fell 9% (test). After viewing the details, 18% of variant users went on, against 75% in the original. The follow-up kept the drop-down and clarified the copy, and click-through rose 9% (test).
- Paperstone's bulk-discount ribbon in the sidebar produced no change in conversions (test). The same hypothesis, redesigned to be more visible, delivered +16.85% revenue and +18.94% AOV, with over 10,000 visitors per version (test).
- Convertibles reran a whisky subscription brand's review test after a four-element bundle lost. The narrower version, a rating row under the price plus a review carousel, moved conversion from 4.725% to 5.460% (+15.56%) over 15,615 visitors and 795 orders (test).
None of those wins would exist on a roadmap that moved straight to the next idea after a loss.
This is iterative testing, and it needs calendar space.
Conversionista
ConvertiblesConversionista
Clean CommitStep 4: Lay out the 13 weeks
A simple quarter for a store with two testable pages and a cart might look like this:
| Weeks | Product page (high traffic) | Cart / checkout | Landing pages |
|---|---|---|---|
| 1-2 | Test 1: bold layout change | QA and setup | Test A: offer framing |
| 3-4 | Test 2 | Test 1: cart drawer change | Test A continues |
| 5-6 | Test 3 | Test 1 continues | Test B |
| 7-8 | Reserved: follow-up slot | Test 2 | Test B continues |
| 9-10 | Test 4 | Reserved: follow-up slot | Reserved: follow-up slot |
| 11-13 | Reserved: follow-up slot | Test 3 | Quarter review |
That is nine planned tests plus four reserved follow-up slots, 13 in all, a little above the median company's pace of nine or ten a quarter.
The four reserved slots are held back for whatever the first half of the quarter teaches you.
Turning it into a CRO roadmap the business will follow
A CRO roadmap fails less often from bad ideas than from losing the room. These habits keep it alive.
Write the decision rule before launch
For each slot, write down what you will do on a win, a loss and a flat result.
A flat result on a change the team wanted anyway can mean "ship it".
A flat result on a costly build usually means "don't".
Clean Commit's checkout-first minicart is a good example of why the rule matters.
Removing recommendations and the chat widget cut conversion 9.9% and revenue per visitor 18.2% over a 7-day test, with a 77.6% probability that control was better (test).
The test was short, so the next slot should be a longer rerun or a version that keeps the recommendations. It should not be a quiet rollout.
Report the program, not single tests
Share three numbers each month: tests launched against plan, share of tests that reached a decision, and the lift from shipped winners. Win rate alone punishes bold tests.
Be careful with the lift number. Optimizely found most uplifts decay to about 80% of their first-month value after a year, while revenue uplifts kept 91% (same report).
Projected annual revenue from a two-week test is an upper bound, not a forecast.
Feed every result back into the backlog
Log each test with its hypothesis, screenshots, numbers and what you would try next in a learning repository.
At the end of the quarter, rescore the backlog with what you learned before you draft the next roadmap.
Before you add an idea, search our test database for others who have tried it.
Treat what you find as evidence for the hypothesis, not a promise.
Our database is skewed toward published wins: about 84% of its 1,724 results are listed as won, far above the 12% platform-wide figure, because companies publish their successes more often than their failures.
For the methods behind each step, start at our A/B testing guide.