A feature flag is a switch in your code that turns a feature on or off for chosen users without a new deploy.
An A/B test is a measurement method: it randomly splits users between versions and compares an outcome metric.
A flag controls who sees what; an A/B test tells you whether it helped.
Most product experiments today run through a flag.
What a feature toggle is (and the four kinds)
"Feature flag" and "feature toggle" mean the same thing.
The code checks a condition at runtime, such as if (flags.isOn("new-checkout")), and runs one code path or the other.
The flag's value lives in a config service, so you can change it for a user, a segment, or a percentage of traffic without shipping code.
The most-cited breakdown comes from Pete Hodgson's 2017 article Feature Toggles (aka Feature Flags) on martinfowler.com.
It sorts toggles by how long they live and how often their value changes:
| Toggle type | What it's for | How long it lives | How dynamic |
|---|---|---|---|
| Release toggle | Hide unfinished work so you can merge and deploy it | Days to weeks | Static |
| Experiment toggle | Split users between code paths and compare outcomes | Hours to weeks | Decided per request |
| Ops toggle | Kill switches and load shedding | Mostly short; some kill switches stay | Needs fast changes |
| Permission toggle | Premium features, beta groups, internal users | Years | Decided per user |
Only one of the four is an A/B test.
Hodgson describes experiment toggles this way: "By tracking the aggregate behavior of different cohorts we can compare the effect of different codepaths."
The other three change who sees a feature but don't measure anything.
Feature flag A/B testing: how a flag becomes an experiment
A flag turns into an A/B test once three things are added: random assignment that stays the same for each user, an exposure log, and a metric you compare between groups.
- Stable assignment. GrowthBook's feature-flag experiment docs explain that the SDK hashes the experiment's seed together with a user ID, so "as long as the user has the same value for this attribute, they will always get assigned the same variation."
- Exposure tracking. In GrowthBook a user counts as enrolled when the app evaluates the flag and they qualify, and that is when a tracking callback logs the exposure. Users who never reach the flagged code are left out of the analysis, and they should be.
- A metric comparison. The exposure log is joined to conversion, revenue, or retention events and analyzed like any other test. The same statistics apply: significance, power, and a planned test duration.
Vendors draw this line in their own products.
Optimizely Feature Experimentation separates A/B test rules from targeted delivery rules on the same flag.
An A/B test rule sends a decision event to the results, while a delivery sends none, because a delivery is just a rollout.
Statsig's feature gate docs say a gate's rollout can be read as a simple A/B test, with the users who get the feature treated as the treatment group.
For more specific hypotheses, the docs point you to a full Experiment. (Since May 2026, Amplitude has run Statsig.)
Because the flag is evaluated in your own code, these tests are usually server-side.
That means no flicker, and you can test things a visual editor can't touch: pricing logic, algorithms, backend data, and in-app flows.
What flag-based tests look like in our database
Many of the SaaS and app tests in our database changed the product itself, not a landing page.
Those changes can only be shipped behind a flag.
- AURUM gave Astrea trial users legal clippings from before they signed up, pulled from an external API, so new users had real data on their first login. Before that, they waited up to 48 hours. The share of trial users engaging with a clipping rose by 16 percentage points (+125% relative). The source describes it as a backend change run as a feature-flag rollout.
- AURUM also added a floating onboarding checklist to the Astrea trial dashboard. Activation rose +187%, and key setup events rose 21% and 25%. This one was a controlled rollout rather than a strict A/B split, so the number is less clean than a randomized result.
- Citrix added an in-product guidance layer to the GoToWebinar free trial and reported 2x trial-to-paid conversions and 40% more product usage.
Feature flag vs A/B test: the differences that matter
The short answer: a flag is plumbing and an A/B test is a measurement.
You can have either one without the other.
| Feature flag (rollout) | A/B test | |
|---|---|---|
| Main question | Is it safe to turn this on? | Did this change the metric? |
| Assignment | Any rule: internal staff, a country, 5% then 50% then 100% | Random, fixed split, sticky per user |
| Comparison group | Often none, or whoever is still on the old version | A control group taken from the same population at the same time |
| Success measure | Errors, latency, crash rate, support tickets | A primary metric chosen before launch, plus guardrails |
| Duration | Until fully rolled out, or forever (permission toggles) | A planned sample size, then stop |
| Who usually runs it | Engineering | Product, growth, or CRO |
| End state | Feature at 100%, then the flag is removed | A decision: ship, iterate, or drop |
A gradual rollout is not a test by itself.
If 5% of users get a feature in week one and 100% in week three, comparing week one to week three mixes the feature's effect with seasonality, marketing, and everything else that changed.
That is a before/after comparison, and our database labels those results as weaker evidence.
Patrick McKenzie's 2008 Bingo Card Creator redesign shows the risk.
The redesign launched to everyone, with no split.
Weekday trial downloads went from about 100 to 125-150, but sales fell from about 2 a day to slightly under 1 (full result).
Without a control group, McKenzie couldn't tell whether the drop came from the redesign or normal fluctuation.
Four years later McKenzie A/B tested a full redesign across the whole site.
Trial signups on AdWords pages went from 20.9% to 23.1%, and purchases were flat at 50 vs 46 (full result).
The split showed something a rollout would have hidden: the gains early in the funnel didn't turn into sales.
When to use a flag, a test, or both
Use a plain flag (no test) when the change is a bug fix, a legal or security requirement, an infrastructure migration, or a feature you're shipping no matter what.
In those cases you only need to know whether it breaks anything, so watch error rates and roll back if needed.
Optimizely's own guidance is to use a rollout when you already know what you're deploying.
Run an A/B test when the change is meant to move a business metric and you have enough traffic to detect a realistic effect.
If you don't, see low-traffic testing and the minimum detectable effect before you commit weeks of traffic to a test you can't read.
Combine them by ramping safely, then measuring. Research from large experimentation teams describes this pattern:
- LinkedIn's Ya Xu, Weitao Duan and Shaochen Huang wrote that most experiments "ramp" traffic gradually, and that teams either "ramp too slowly and much time and resource is wasted; or we ramp too fast and suboptimal decisions are made." Their SQR framework splits ramping into four phases that balance speed, quality and risk.
- Aleksander Fabijan's Safe Velocity paper (ICSE-SEIP 2019) describes "controlled rollout," which applies a controlled experiment to each ring of a phased rollout. It was evaluated across hundreds of Microsoft Office deployments.
In practice: release to internal users first, then a small random slice as a canary to catch crashes.
Then open a 50/50 experiment rule and keep the split fixed until you reach the planned sample size.
If you want the feature on permanently but still want to track its long-term effect, keep a small holdout group without it.
The OnSwitch result shows why the measuring step matters even for changes that look like a win.
The freemium lighting app locked some popular free scenes for the first three days, then unlocked them with a push notification.
Users came back slightly more but bought less, and a repeat with different locked content gave similar results.
A rollout that only watched return visits would have called it a success.
Mistakes specific to flag-based tests
- Changing the split mid-test. GrowthBook warns that "changing the split percentages mid-experiment risks having a user switch variations." Ramp before the experiment starts, or start a new phase, and check for sample ratio mismatch.
- Logging exposure in the wrong place. If you log every user at login but only some reach the flagged screen, you dilute the effect. If you log only after an action that the treatment itself affects, you bias the groups.
- Overlapping flags. Several experiments on the same screen can interfere with each other. See interaction effects for when that matters.
- Skipping QA on each variation. Every flag state is a code path a real customer will hit. Use a QA checklist before you open traffic.
Flag debt: removing toggles after the decision
Hodgson's article advises treating flags as inventory: "Savvy teams view their Feature Toggles as inventory which comes with a carrying cost and seek to keep that inventory as low as possible."
Each live experiment flag doubles the paths through that part of the code.
The standard warning is Knight Capital. According to the SEC, on August 1, 2012, Knight's router sent more than 4 million orders in the first 45 minutes of trading while trying to fill 212 customer orders, and the firm lost more than $460 million.
The cause was old, unused code that a faulty deployment triggered.
Knight later paid a $12 million SEC penalty.
Hodgson cites it as what can go wrong "when you don't manage your feature flags correctly."
A simple rule for experiment flags: when the test ends, write down the decision, set the winner to 100%, and open a ticket to delete the losing code path.
Record what you learned in a learning repository so the result outlives the flag.
If you're choosing a platform that does both, see our reviews of GrowthBook, Statsig and Optimizely, or start from the A/B testing guide.