A CRO glossary defines the terms people use when they try to get more sales, sign-ups or revenue from the same traffic: conversion rate, lift, sample size, variance and the rest.
Every term below has a one-line definition and a link to its own page, and the statistics and methods terms link to the full guide.
Where it helps, we show a term at work in a real result from our database of A/B tests.
New to the field? Start with what conversion rate optimization is and how A/B testing works, then use this page when a word trips you up.
A/B testing glossary: terms A to Z
Each entry links to a short page with the definition, a formula where there is one, and test results from our database that show the term in practice.
B
- Banner blindness: visitors skip over content that looks like an ad or sits where ads usually go, even when it is useful.
- Baseline conversion rate: the conversion rate of your current page before a test. It is the first input to any sample size calculation.
C
- Canary release: shipping a change to a small share of users first, so problems show up before everyone gets it. It is a safety check, not a controlled experiment.
- Click-through rate (CTR): clicks divided by impressions or views, shown as a percentage.
- Control and variant: the control is the unchanged version in a test; the variant (or treatment) is the changed version measured against it.
- Conversion rate: conversions divided by visitors or sessions. It is the headline CRO metric, but it ignores order size.
- Customer lifetime value (CLV or LTV): the revenue or gross profit you expect from a customer over the whole relationship, not just the first order.
E
- Experimentation: the practice of deciding product and marketing changes with controlled tests rather than opinion.
F
- F-pattern: a scanning pattern seen in eye-tracking studies of text-heavy pages, where readers scan across the top, then a shorter line lower down, then down the left side.
- Friction: anything that makes an action harder than it needs to be, such as extra form fields, surprise costs or slow pages.
H
- Hero section: the large first block of a page, usually a headline, image and main call to action.
I
- Information scent: the cues, such as link labels and menu names, that tell visitors whether a path leads to what they want.
L
- Landing page: the page a visitor arrives on from an ad, email or search result, often built around one offer and one action.
- Lift: the relative change between variant and control. A move from 2.0% to 2.2% conversion is a 10% lift, not 0.2%.
N
- Net Promoter Score (NPS): the percentage of customers who score you 9 or 10 on "how likely are you to recommend us," minus the percentage who score 0 to 6.
P
- Painted door test: showing a button or offer for something that does not exist yet, to measure demand before you build it.
S
- Sample size: the number of visitors each group needs before you judge a test. It depends on baseline rate, minimum detectable effect, significance level and power.
T
- Trust signals: page elements that reduce doubt about buying, such as reviews, guarantees, security badges and expert endorsements.
U
- Unique visitors: the count of distinct people (in practice, distinct browsers or devices) who visited in a period, each counted once.
- UTM parameters: tags added to a link, such as utm_source, utm_medium and utm_campaign, that tell your analytics tool where a visit came from.
V
- Variance: how spread out a metric is around its average. Noisier metrics, such as revenue per visitor, need bigger samples to show a real difference.
Experimentation glossary: terms with a full guide
Some terms need more than a definition, so they get a full guide instead of a term page.
Here is each one in a line, grouped by topic.
Statistics
| Term | In one line |
|---|---|
| Statistical significance | A result unlikely to be random noise at the confidence level you set. It says nothing about whether the effect is large. |
| p-value | The chance of seeing a difference at least this big if the change did nothing. It is not the chance the variant is better. |
| Statistical power | The chance a test detects a real effect of a given size. 80% is the usual target. |
| Minimum detectable effect (MDE) | The smallest lift a test is sized to detect reliably. Halving it roughly quadruples the traffic you need. |
| Confidence interval | A range of plausible values for the true effect, not just a single number. |
| Bayesian vs frequentist | Two ways to read a test: probability to beat control, or p-values against a fixed sample size. |
| Sequential testing | Methods that let you check results as data arrives without inflating false positives. |
| CUPED | A variance-reduction method that uses pre-test data to reach a result with less traffic. |
| Multiple comparisons | The rising false-positive risk when you test many variants, metrics or segments at once. |
Test types and setup
| Term | In one line |
|---|---|
| A/A test | Both groups see the same page, to check that your tool splits traffic and reports correctly. |
| Multivariate test | Tests every combination of several changes at once. Needs far more traffic than an A/B test. |
| Split URL test | Each version lives at its own URL, common for full redesigns. |
| Multi-armed bandit | Shifts traffic toward the better version while the test runs. Good for earning, weaker for learning. |
| Holdout group | Users kept out of a change for months to measure its long-run effect. |
| Feature flag | A switch in code that turns a feature on for chosen users, often used to run tests and canary releases. |
Things that break a test
| Term | In one line |
|---|---|
| Sample ratio mismatch (SRM) | The traffic split differs from what you set by more than chance allows, which usually means a bug. |
| Novelty effect | An early lift from newness that fades over time. |
| Flicker effect | The original page flashes before the variant loads in client-side tests. |
| Winner's curse | Winning tests tend to overstate the true lift, so shipped results often come in smaller. |
| Regression to the mean | Extreme early numbers drift back toward average as more data comes in. |
| Simpson's paradox | A trend in every segment reverses when the segments are combined. |
| Guardrail metric | A metric you watch so a winner does not cause harm elsewhere, such as refunds or cancellations. |
For the psychology terms (anchoring, loss aversion, the decoy effect and more), see our guide to cognitive biases in CRO, which separates what lab research shows from what A/B tests show.
Glossary terms in real test results
Definitions make more sense next to real numbers. Three examples from the database:
- Novelty effect and early peeking. Ruben added a doctor or expert endorsement badge to a product page and saw +17% conversion rate after one week (99.6% probability). The test is logged as flat overall, which is why one strong week is not a result.
- Revenue per visitor vs conversion rate. Dave Diederen removed a free-shipping threshold and charged shipping on every order, and recorded +9.17% revenue per visitor. Charging for shipping is the kind of change a conversion-rate-only view can misread.
- Segments. Dave Diederen's countdown timer for a European market produced -35.61% revenue per visitor, while an order-reserved checkout timer for US brands gained +4.83% revenue per visitor. The same idea can win in one audience and lose in another.
- Trust signals. Kasper Dolk tested a testimonial against no testimonial and saw revenue per visitor rise 7.10% (from €5.36 to €5.74).
A
→B
+9.17% revenue per visitor
Removed the free-shipping threshold and charged shipping on every order
Site-wideShipping & delivery messaging
@DaveDiederen
@DaveDiederenA
→B
+4.83% revenue per visitor
Added a checkout countdown timer (order reserved) for US brands
CheckoutCountdown timers@DaveDiederen
@DaveDiederenA
→B
Revenue per visitor +7.10% (€5.36 to €5.74)
Show a testimonial vs no testimonial
OtherReviews & star ratings
@kasper_dolk
@kasper_dolkA
→B
-35.61% revenue per visitor
Added a countdown timer for a non-US (European) market
Site-wideCountdown timers@DaveDiederen
@DaveDiederenA
→B
+17% conversion rate after 1 week (99.6% probability)
Added an authority badge (doctor/expert endorsement like FrontrowMD) to the page
Product pageSocial proof, authority & press
@Rubendgrf
@RubendgrfTo see more terms in action, browse every A/B test result in the database by page type and element, or read how we collect and check results.