Canary testing (a canary release) is a way to ship a change safely: you send the new version to a small slice of users first, watch for errors or drops in key metrics, and only then roll it out to everyone.
Its job is to catch problems early, not to prove that the new version is better.
That second question belongs to an A/B test.
The name comes from the canaries miners carried underground.
The bird reacted to toxic gas before the miners did, so it worked as an early warning.
Danilo Sato's 2014 write-up on martinfowler.com defines the technique as slowly rolling a change out "to a small subset of users before rolling it out to the entire infrastructure."
Canary release in CRO and A/B testing
A canary release and an A/B test look alike on the inside.
Both split traffic and both usually run through feature flags.
The difference is the question each one answers.
| Canary release | A/B test | |
|---|---|---|
| Question | Is anything broken? | Is the variant better than the control? |
| Traffic split | Small slice first (for example 1-5%), then ramped up | Fixed split, often 50/50, held to the end |
| Typical length | Minutes to hours per step | Days to weeks, until the planned sample is reached |
| What you watch | Errors, latency, crashes, sharp metric drops | A primary conversion metric plus guardrails |
| Outcome | Roll forward or roll back | Ship, iterate or drop the idea |
Sato's article makes the same point: the two share an implementation but should stay separate, because A/B tests usually need days to reach significance while a canary rollout should finish in minutes or hours.
A 5% canary that looks fine after an hour tells you the code works.
It tells you almost nothing about whether conversion went up.
Where the ideas meet is the guardrail metric.
A good canary watches revenue per visitor or checkout completion alongside error rates, so a release that runs cleanly but quietly hurts sales still gets caught.
Blue green deployment
Blue green deployment is the all-at-once cousin. As Martin Fowler described it in 2010, you keep two production environments that are as identical as possible.
One (blue) is live, you release to the other (green), then switch the router so all traffic goes to green.
If something goes wrong, you switch the router back.
So blue green swaps 100% of traffic in one move with a fast way back, while a canary moves traffic in small steps.
Teams often combine them: the green environment receives a canary slice first, then the rest.
Neither method measures lift. Both decide whether a release is safe, not whether it helps.
If a change matters to revenue, run it as an A/B test with a held-out control before or during the rollout.
Example: why redesigns need more than a canary
Full redesigns show why. None of these was a technical failure, so a canary watching only technical health would likely have passed them all.
It took measuring conversion against the old version to expose what they did:
- Clean Commit tested a single-row header with an image mega-menu against the old navigation: -22.5% conversion.
- Clean Commit's product page redesign lost -16.3% conversion rate and -12.2% revenue per visitor.
- Patrick McKenzie's full site redesign and merger brought more trials, but sales roughly halved.
Clean CommitClean CommitA practical pattern: canary the release at a few percent to confirm nothing breaks, then hold it at a fixed split as a proper A/B test until it reaches its planned sample.
Ramp to 100% only if the business metric holds up.
Related terms
- Feature flags: the switch most canary releases and server-side tests run on.
- Guardrail metrics: the "do no harm" checks a canary should include.
- Control and variant: the comparison a canary lacks and an A/B test provides.
- More definitions in the glossary, and real results in the test database.