Guardrail metrics are the numbers an A/B test must not hurt, such as revenue per visitor, refunds, page speed or trial-to-paid rate, even though the test is not trying to move them.
The primary metric decides whether a variant wins; guardrails decide whether it is safe to ship.
Both should trace back to one overall evaluation criterion (OEC) that reflects long-term value, which is the experiment-level version of a company's north star metric.
This page sits under our A/B testing metrics guide.
It covers how the four layers fit together, then shows what happens in real tests from our database when a team watched only one number.
North star metric: the goal every test serves
Amplitude, whose North Star Playbook popularized the framework, defines a north star metric as "a single metric that best captures the value customers derive from your product."
Its three qualities: it represents the value users get, it sits within product and marketing's sphere of influence, and it is a leading indicator of revenue.
Amplitude credits the term to Sean Ellis and the growth-hacking community.
A north star is usually too slow and too broad to decide a single test.
"Weekly active users who got an answer" will not move measurably because you changed a button color.
So teams break it into input metrics they can affect, and pick experiment metrics from those inputs.
The north star still matters in testing because it tells you which test results count.
A test that lifts clicks but works against the north star is not a win, no matter how significant the click lift is.
Primary metric: the one number that decides the test
Optimizely's documentation says the primary metric "determines whether the test wins or loses."
You pick it before launch, size the test for it, and judge the result on it. Everything else is context.
A good primary metric is close enough to the change to move, but connected enough to the business to matter.
Pick something too close to the change and you get lifts that never reach revenue.
Clean Commit's one-click size buttons on collection cards raised add-to-cart by 29.7%, yet conversion rate changed 0.0%, orders fell 3.5% and revenue per visitor fell 10.4% across 13,432 desktop visitors.
Judged on add-to-cart, it was a big winner. Judged on sales, it was not.
The same pattern shows up with engagement metrics.
Blend Commerce's sticky mobile filter bar lifted filter and sort clicks by 299.58%, while conversion rate moved -2.23% and RPV -1.80% (not significant).
The store shipped it as a UX change, not as a revenue win.
The opposite mistake is choosing a metric so far downstream that the test can never reach significance.
Ron Kohavi and colleagues give a worked example in their 2009 survey of online controlled experiments, using the rule-of-thumb sample size n = 16 × variance / (change to detect)², per variant, for 95% confidence and 80% power:
| Primary metric | Inputs | Users needed per variant to detect a 5% change |
|---|---|---|
| Revenue per user | Mean $3.75, standard deviation $30 | 16 × 30² / (3.75 × 0.05)² ≈ 409,600 |
| Conversion rate | 5% purchase rate, variance 0.05 × 0.95 | 16 × 0.0475 / (0.05 × 0.05)² ≈ 121,600 |
Conversion needs about 3.3 times fewer users, which in the paper's example cuts a six-week test to two weeks.
That is why many teams use conversion as the primary metric and revenue per visitor as a guardrail.
If you have the traffic, our revenue per visitor guide explains when RPV should be the primary instead.
Read statistical power and minimum detectable effect before you choose.
Overall evaluation criterion: what "better" means
Kohavi's paper defines the OEC as "a quantitative measure of the experiment's objective," also called the response, dependent variable, outcome or evaluation metric.
The authors recommend a single metric, possibly a weighted combination of objectives, because it "forces tradeoffs to be made once for multiple experiments and aligns the organization behind a clear objective."
They also warn that a good OEC "should not be short-term focused (e.g., clicks)" and should include factors that predict long-term goals, such as predicted lifetime value and repeat visits.
The paper adds that the OEC must be chosen in advance.
Picking the metric after you see the data raises the chance of a false win, the same problem covered in multiple comparisons.
Several tests in our database only make sense with a long-term OEC:
- Zumba sent paywall users to a web checkout. Immediate conversions fell 25%, but LTV rose 17%. Judged on conversion, it lost; judged on lifetime value, it won.
- Server Density moved to packaged pricing tiers. Free sign-ups fell 24.96% (6.92% to 5.19%), while revenue rose 114% ($394 to $833). The paid counts were small, 20 vs 15, so treat it as directional.
- BoldVoice tested a hard paywall with no close button. Trial starts rose, but refunds rose by more than the gain. The team now judges tests on net revenue per exposed user on matured cohorts.
For most stores, a practical OEC is revenue or gross profit per visitor.
For subscription businesses, it is usually net revenue per user after refunds, measured on a cohort old enough to have renewed or churned.
Guardrail metrics: what the test must not break
A guardrail is a metric you don't expect to improve but would refuse to ship if it got worse.
Optimizely calls these monitoring goals, which "form a warning system that alerts you when you hurt another revenue path."
Real tests show why. Hubstaff's homepage redesign lifted visitor-to-trial conversion by 49% over three months and 259k visitors, but trial-to-paid fell 7%.
The team attributed the drop to factors outside the homepage.
Without trial-to-paid as a guardrail, nobody would have had that conversation.
Clean Commit's best-sellers 404 page raised conversion rate 6.8%, but AOV fell 6.0%, so RPV rose only 0.3%.
Guardrails usually fall into three groups:
| Type | Examples | Why it matters |
|---|---|---|
| Business | RPV, AOV, gross margin, refunds, trial-to-paid, lead quality, unsubscribes | Catches wins on the primary metric that cost money elsewhere |
| User experience | Page load time, error rate, support tickets, bounce rate | Catches damage the primary metric may not show during the test window |
| Trust (data quality) | Sample ratio mismatch, tracking coverage, bot share | If these fail, no other metric in the test can be trusted |
Speed is the classic UX guardrail. Kohavi's paper reports that Microsoft Live Search experiments that slowed the results page by one second cut queries per user by 1% and ad clicks per user by 1.5%.
At two seconds, the drops were 2.5% and 4.4%.
If speed is not in your OEC, check that a losing variant is not losing simply because it is slower.
Clean Commit
ConvertiblesClean CommitHow to judge a guardrail statistically
The most common mistake is reading "no significant drop" as "safe."
A test sized for a 5% conversion lift is often far too small to detect a 3% drop in RPV, so the guardrail passes simply because it lacked power.
Spotify's experimentation team handles this with non-inferiority tests.
Their decision rule, described in a 2024 paper by Mårten Schultzberg, Sebastian Ankargren and Mattias Frånberg, is to ship when the treatment is significantly superior on at least one success metric and significantly non-inferior on all guardrail metrics.
Each guardrail gets a margin you would accept, for example "RPV may not fall more than 2%."
Two corrections follow from that rule:
- False positives: Spotify does not adjust alpha for guardrails, because every guardrail must pass. Only success metrics need correction, alpha / S, where S is the number of success metrics.
- False negatives: Power does need correcting. Spotify sets beta per metric to beta / (G + 1), where G is the number of guardrails. With five guardrails each at 80% power, the chance that all pass together drops below 40%.
Here is a worked example with illustrative numbers.
Your guardrail is RPV with a 2% non-inferiority margin.
The variant shows RPV down 1.0%, with a standard error of 1.2 percentage points on the relative difference.
| Scenario | Observed change | Standard error | One-sided 95% lower bound (change − 1.645 × SE) | Result |
|---|---|---|---|---|
| Small test | -1.0% | 1.2 pts | -2.97% | Fails: the drop could be worse than 2% |
| Larger test | -1.0% | 0.5 pts | -1.82% | Passes: non-inferior within the 2% margin |
The point estimate is the same in both rows.
Only the precision changed. A one-sided 95% bound is the lower end of a two-sided 90% confidence interval.
Airbnb uses a related setup. According to InfoQ's report on Airbnb's Experiment Guardrails system, it checks three things: impact (the metric must not fall below an escalation threshold t), power (the standard error must be small enough, for example below 0.8 × t), and a "stat sig negative" check for sensitive metrics like revenue.
The system flagged about 25 experiments a month, and roughly 5 of those were stopped after review.
Optimizely takes a different route. It gives each monitoring goal a weight of 1/n in its false discovery rate control, where n is the number of monitoring metrics, so adding guardrails barely slows the primary metric.
When guardrails mislead
- Too many guardrails. Every one you add lowers the chance that a truly good variant passes all of them. Keep three to five that would actually change your decision.
- Segment guardrails without correction. Blend Commerce's Vegepod accordion test lost overall (-8% RPV) but won on desktop (+26% RPV) and lost on mobile (-28% RPV). Device splits like this are worth checking, but treat any single segment result as a lead to retest, not a verdict.
- Short-window guardrails for long-term harm. Refunds, churn and lead quality show up weeks later. Use matured cohorts or a holdout group for these.
- A guardrail treated as a second primary. If you would ship a variant because the guardrail rose, it is really your OEC. Write that down before launch.
A starter set by business model
| Business | Typical primary | OEC | Guardrails |
|---|---|---|---|
| Ecommerce | Conversion rate | Revenue or gross profit per visitor | RPV, AOV, return rate, page speed, SRM |
| SaaS | Visitor-to-trial or sign-up rate | Paid conversions or revenue per visitor | Trial-to-paid, activation, support tickets |
| Lead gen | Form submissions | Qualified leads or pipeline per visitor | Lead quality, cost per acquisition, spam rate |
| Mobile app | Trial start rate | Net revenue per user on matured cohorts | Refunds, trial-to-paid, cancellations, ratings |
Then browse the test database for results in your vertical and note which ones reported a guardrail at all.
The ones that did, such as Convertibles' shipping protection toggle (conversion 1.53% to 1.90%, +16.4% RPV, AOV -6%), are the ones you can trust most.
For the full process, see how to analyze A/B test results.