An A/B test repository is a searchable record of every experiment a team has run, including the losers and the flat results, with the hypothesis, the exact change, the numbers and what was learned.
It stops teams from re-running old tests, lets them spot patterns across many experiments, and turns single results into knowledge that outlasts the people who ran them.
Without one, a test result lives in a slide deck or a chat thread and is gone within a year.
With one, the next person to propose "let's add a countdown timer" can see what happened the last five times someone tried it.
What an experiment repository is and why it matters
An experiment repository (also called a test library or learning repository) is a single place where each completed test gets a structured record.
It is different from your testing tool's dashboard, which shows raw results but rarely captures why you ran the test or what you decided afterward.
Ron Kohavi, Diane Tang and Ya Xu give the idea a full chapter, "Institutional Memory and Meta-Analysis," in Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).
Their point: as a program grows, the history of past experiments becomes one of its most valuable assets, because it shows which patterns hold across tests and which do not.
Why one result is not enough: the countdown timer record
Countdown timers show why a single test result can mislead.
Here is what several stores in our database found when they tested a timer at checkout or across the site:
| Test | Who ran it | Result |
|---|---|---|
| Checkout "order reserved" timer, US brands | Dave Diederen | +9.50% conversion rate, +4.83% revenue per visitor, -4.26% AOV |
| Countdown timer added to checkout | Dave Diederen | +3.16% revenue per visitor, +2.99% conversion rate |
| Countdown timer added to checkout (a later test) | Dave Diederen | -7.1% revenue per visitor, -5.4% conversion rate |
| Countdown timer for a European market | Dave Diederen | -35.61% revenue per visitor, -17.35% conversion rate |
| Removed the timer from a Dutch store's checkout | Kasper Dolk | +25.61% conversion rate, +26.18% revenue per visitor from removing it |
Read one of these results alone and you get a confident, wrong rule.
Read them together and a pattern shows up: the timer tended to help some US stores and hurt in European markets.
In Kasper Dolk's case, a post-purchase survey said the timer felt like too much pressure, and the test confirmed it.
That is the job of a repository. It turns "timers work" or "timers don't work" into "timers worked for these audiences on these pages, and here is the evidence."
It also stops you from giving up on good ideas
A repository should keep losing tests next to the tests that came after them.
Conversion Fanatics redesigned a luxury skincare kit page and the first version lost by a large margin.
They kept the record, reworked the idea with radio buttons, a benefit headline and a "Buy Now" button, and the next version won with +35.4% sales and +41% revenue per visitor (the post notes that test was still running when it was published).
Without the record of the loss, the second test either never happens or repeats the first one's mistakes. This is the core of iterative testing.
Meta-analysis of A/B tests: what a large library can tell you
A meta-analysis pools many experiments to estimate the typical effect of a type of change.
Once your repository has enough tagged tests, you can run a simple version yourself.
Three published meta-analyses show what this looks like at scale.
What large meta-analyses have found
- Qubit, 6,700+ experiments (2017). Will Browne and Mike Swarbrick Jones grouped Qubit's ecommerce tests into 29 treatment types. Scarcity (stock pointers) averaged +2.9% revenue per visitor, social proof +2.3% and urgency +1.5%, while color changes averaged 0.0% and calls to action -0.3%. They also found 90% of experiments moved revenue by less than 1.2% in either direction.
- Wharton, 2,732 tests from 252 companies. Alex Miller and Kartik Hosanagar studied ecommerce A/B tests across seven industries over three years. Price promotions and tests on category and product listing pages had the largest effects, and price promotions worked best early in the funnel while shipping promotions worked best late.
- Analytics Toolkit, 1,001 tests (2022). Georgi Georgiev analyzed tests run on the Analytics Toolkit platform, screening for outcome-unbiased data. 33.5% had a statistically significant winner, the median lift across all tests was 0.08%, and the median test ran 30 days.
Notice the spread in the Qubit data. Urgency averaged +1.5%, but with a standard deviation of 2.8% across treatments.
That matches the countdown timer table above: the average hides results that point in opposite directions.
The authors noted that urgency may behave differently depending on what it counts down to, but they did not have enough tests to say for sure.
Publication bias: the trap in every shared library
Public test collections, including ours, lean toward winners because people share wins more than losses.
Of the 1,724 results in LiftArchive, 1,447 are listed as won, about 84%.
Compare that with the 33.5% significant-winner rate in Georgiev's screened sample, or with the program below that publishes everything.
Clean Commit publishes its whole test library. Of its 173 tests in our database, 16 won, 35 lost and 122 were flat, a win rate of about 9%.
Many of those flat results looked exciting before the stats came in: a one-measurement size finder showed +33.1% conversion rate over 20,402 visitors and 21 days, but RPV probability was only 72.5% and AOV fell 14.6%, so it was counted as inconclusive.
Your internal repository is the one place where you can avoid this bias completely.
Log every test, not just the ones that make a good slide.
Our win rate research goes deeper into what honest numbers look like.
When meta-analysis is the wrong tool
- When the tests measured different things. Averaging a revenue-per-visitor lift with an add-to-cart lift gives a number that means nothing. Group by metric first.
- When changes were bundled. A redesign that changed five things at once cannot tell you which one drove the result. Vasta's five trust signals beside add-to-cart won +22.95% conversion rate, but the record itself notes all five were added together.
- When the contexts differ more than the treatments. Convertibles found a trust bar under the buttons on handbag pages cost -10.2% RPV (with 516-547 orders per arm, below their own 1,000-order target, so nothing shipped). A pooled "trust bars help" estimate would not have predicted that for a premium brand.
- When the sample is small. Ten tagged tests will not give a stable average. Use them as a list of things to check, not as a forecast.
- When winners were measured at their peak. Lifts reported at the moment of significance tend to be overstated, which inflates any average built from them. See the winner's curse.
Building a test knowledge base your team will actually use
A test knowledge base fails when entries are hard to write or hard to find.
Keep each record short and structured, and make the tags consistent so you can filter later.
What to record for every test
| Field | What to write | Why it matters |
|---|---|---|
| Hypothesis | The problem, the change and the expected effect | Lets you judge the idea, not just the outcome. |
| Evidence behind it | Survey, heatmap, analytics or usability finding | Shows which research sources produce winners |
| Control and variant | Plain description plus screenshots | Screenshots are what people search for a year later |
| Page, element, audience, market | Consistent tags from a fixed list | Makes filtering and meta-analysis possible |
| Primary metric and guardrails | Metric name, baseline, result, interval | Stops apples-to-oranges comparisons |
| Sample and duration | Visitors or orders per arm, days run | Shows how much to trust the result |
| Outcome | Won, lost or flat, by your pre-set rule | Flat and lost results are half the value |
| Decision and learning | Shipped or not, and one sentence on what you learned | The learning is what the next test builds on |
Habits that keep it useful
- Write the record before launch. Fill in the hypothesis, metric and stopping rule first, then add results. This also guards against changing the success metric afterward.
- Use a fixed tag list. "PDP," "product page" and "product-page" become three different things in a filter. Pick one name for each page type and element.
- Search before you prioritize. Make "check the repository" a step in your prioritization process. Past results are some of the best evidence for scoring an idea.
- Review it on a schedule. A quarterly look across tags shows which areas keep producing wins and which keep coming up flat, which feeds your testing roadmap.
- Re-test old winners. Audiences and markets change. Dave Diederen's two checkout timer tests above point in opposite directions, and the record makes that visible.
Where to keep it
A shared spreadsheet or a Notion or Airtable database is enough for most teams running a few tests a month.
Larger teams often use the experiment archive built into their testing platform, or a dedicated knowledge tool.
Airbnb built and open-sourced one, the Knowledge Repo, which publishes peer-reviewed analyses written in Markdown, Jupyter or R Markdown; its GitHub repository is now archived and read-only.
The tool matters less than the habit. A spreadsheet that every test goes into beats a polished system that only holds the wins.
Use an outside library to fill the gaps
Your own repository only knows what you have tested.
Before you test something new, look at what others found.
LiftArchive's test database lets you filter 1,724 results by page type, element and vertical, each credited to whoever ran it, and our list of best practices that failed collects the cases where common advice did not hold up.
Treat outside results the way you treat your own: as evidence about a context, not a promise.
Then add your own result to your repository, whichever way it goes.
@DaveDiederen
@kasper_dolk
Conversion Fanatics@DaveDiederen@DaveDiederen
Clean CommitFor the bigger picture of running tests well, start with our A/B testing guide.