Experimentation is the practice of testing a change by randomly splitting people into groups, showing each group a different version, and measuring which version performs better on a metric chosen in advance. In online businesses this is called an online controlled experiment, and the most common form is the A/B test.
Online controlled experiments
An online controlled experiment has four parts: a control and at least one variant, random assignment of visitors, a primary metric, and a statistical test that decides whether the gap between groups is larger than chance would produce.
Random assignment is what makes it an experiment rather than a before-and-after comparison.
Because both groups see the site during the same days, with the same traffic mix and promotions, the only systematic difference between them is the change being tested.
Ronny Kohavi, Diane Tang and Ya Xu use the term as the title of their book, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).
The scale can be large: a 2017 ICSE paper by Fabijan and colleagues reports that Bing runs more than 10,000 experiments a year.
Experimentation definition in CRO and A/B testing
In conversion rate optimization, experimentation means treating every page change as a hypothesis to check, not a decision to ship on opinion.
A team writes down what it expects to happen and why, runs the test until it reaches its planned sample, then reads the result against a significance threshold.
The word also covers the wider habit. An experimentation program is the process around single tests: a backlog of ideas, prioritization, QA, analysis and a record of what was learned.
Most ideas do not work. In a 2009 Microsoft paper, Kohavi and colleagues wrote that "only about 1/3 of ideas improve the metrics they were designed to improve," with the rest flat or negative.
That is the main argument for testing: without a control group, the two thirds that do not help would ship too.
Examples from the database
Experiments end in three ways, and our database has all three.
- A win. Kasper Dolk tested a benefits-first product hero against the control hero. Revenue per visitor went from €1.20 to €1.47, a reported +23.05% lift.
- A loss. MarketingExperiments tested a three-step free trial sign-up on the RegOnline homepage. Completed leads fell 24.5%, a change that would have looked reasonable in a design review.
- A flat result. Patrick McKenzie ran a time-limited discount for Bingo Card Creator trial users. The groups converted at 1.87% vs 2.08%, and the result was inconclusive.
A fourth case shows why the metric matters.
Patrick McKenzie's 2012 full site and app redesign lifted trial sign-ups 10%, but purchases stayed flat.
A test judged only on sign-ups would have called it a win.
@kasper_dolk
MarketingExperimentsWhen it is not really an experiment
A before-and-after comparison, where you change the page and compare this month to last month, is not a controlled experiment.
Seasonality, ad spend and traffic mix all change at the same time, so you cannot tell what caused the difference.
Experimentation is also a poor fit when traffic is too low to reach a useful sample in a few weeks, or when the change is required anyway (a legal fix, a broken checkout).
In those cases, ship the change and watch the numbers rather than run a test that cannot answer the question.
Related terms
- Control and variant: the versions being compared.
- Lift: the relative difference between them.
- Sample size: how many visitors each group needs.
- Painted door test: an experiment on demand for something not yet built.
For more terms, see the CRO glossary, or browse all A/B test results.