A two-tailed test checks whether a variant differs from the control in either direction, so it can flag a significant winner or a significant loser.
A one-tailed test only checks one direction, usually "is the variant better?", which gives it more power but makes it blind to harm.
For most A/B tests, use two-tailed unless you decided before launch that a loss and a tie would lead to the same action.
This setting is one of the most-asked questions in any significance calculator.
Below we explain what each test does, show the math with a worked example, and use real results from our test database to show where the choice changes the verdict.
What a one-tailed test is
Every A/B test starts with a null hypothesis: the variant and control convert at the same rate.
The alternative hypothesis says what you are looking for instead.
A one-tailed (one-sided) test picks a single direction, such as "the variant converts higher than the control."
The "tail" is the end of the bell curve where results would count as surprising.
A one-tailed test puts all of your significance threshold (alpha) in one tail.
At alpha = 0.05, the whole 5% sits on the "variant is better" side, so the critical z-score is about 1.645.
The UCLA Statistical Methods and Data Analytics FAQ describes it this way: a one-tailed test "allots all of your alpha to testing the statistical significance in the one direction of interest."
The price is that a large effect in the other direction is never declared significant.
The test can only say "not a winner."
Some A/B testing practitioners argue that one-sided tests fit A/B testing better.
Georgi Georgiev, who runs Analytics-Toolkit, writes that "in an A/B test the hypothesis of interest is usually one-sided", since you only ship a change if it helps.
In a longer article, Georgiev argues that a properly chosen one-tailed test does not raise the false positive rate, and that a two-sided test lets you claim a direction it wasn't designed to test.
What a two-tailed test is
A two-tailed test's alternative hypothesis is "the variant converts differently from the control," higher or lower.
It splits alpha between the two tails. At alpha = 0.05, that is 2.5% on each side, so the critical z-score rises to about 1.96.
Because the bar is higher on each side, a two-tailed test needs a slightly bigger observed difference to call a winner.
In return, it can also call a significant loser, which matters a lot when you are testing changes that might hurt revenue.
This is the default in most commercial tools.
Optimizely's statistical significance docs say two-tailed tests "tell you if your variation is a winner and if your variation is a loser," and that Optimizely uses them because "they are required for the false discovery rate control in Stats Engine."
One-tailed vs two-tailed test: the math side by side
For conversion rates, both tests usually use the same z-statistic.
Only the way you turn it into a p-value changes.
- z = (p_B - p_A) / SE
- SE = sqrt( p_pool x (1 - p_pool) x (1/n_A + 1/n_B) ), where p_pool is total conversions divided by total visitors
- One-tailed p-value (testing "B is better"): p = 1 - Phi(z)
- Two-tailed p-value: p = 2 x (1 - Phi(|z|))
Phi is the standard normal cumulative distribution. When the result falls in the predicted direction, the one-tailed p-value is exactly half the two-tailed one.
When it falls the other way, the one-tailed p-value is 1 minus half the two-tailed one, as the UCLA FAQ notes.
| One-tailed (B > A) | Two-tailed (B ≠ A) | |
|---|---|---|
| Alternative hypothesis | Variant converts higher | Variant converts differently |
| Alpha placement at 0.05 | 5% in the upper tail | 2.5% in each tail |
| Critical z at 95% | 1.645 | 1.96 |
| Critical z at 90% | 1.282 | 1.645 |
| Can declare a significant loser? | No | Yes |
| Sample size at 95% significance, 80% power | Baseline | About 27% more visitors |
| Sample size at 90% significance, 80% power | Baseline | About 37% more visitors |
The sample size rows are our calculation from the standard formula, where required n scales with (z_alpha + z_beta)^2. At 80% power, z_beta is 0.842. So the ratio at 95% is (1.96 + 0.842)^2 / (1.645 + 0.842)^2 = 1.27. Georgiev's article gives a similar range of 20-60% more sample for two-sided tests, depending on the threshold.
Our sample size calculator covers the rest of that formula.
Worked example: same data, two verdicts
Say each arm gets 10,000 visitors. The control gets 300 conversions (3.00%) and the variant gets 345 (3.45%), a 15% relative lift. These numbers are made up for illustration.
- Pooled rate: 645 / 20,000 = 3.225%.
- SE = sqrt(0.03225 x 0.96775 x (1/10,000 + 1/10,000)) = 0.00250.
- z = (0.0345 - 0.0300) / 0.00250 = 1.80.
- One-tailed p = 0.036. Significant at 0.05.
- Two-tailed p = 0.072. Not significant at 0.05.
Now flip it. The variant gets 240 conversions (2.40%), a 20% drop.
Here z = -2.62. The two-tailed p-value is 0.009: a clear, significant loser.
The one-tailed "B is better" p-value is 0.996. That test simply reports "no evidence the variant is better" and says nothing about the loss.
Real tests where the choice mattered
The one-tailed vs two-tailed choice is not just a classroom point. It shows up in published results.
GrowthRock tested a category link bar on an apparel store's mobile homepage.
Over 28 days, 80,000+ visitors and about 2,300 conversions per variation, the variant showed +5% orders at 93% significance one-tailed.
By Optimizely's two-tailed stats it was not significant.
GrowthRock called it a likely winner, and we list it as inconclusive.
Optimizely's numbers also come from sequential testing, so the tails are not the only difference, but this is the gap the worked example above shows.
Losers are where two-tailed tests earn their keep.
Clean Commit's single-row header with an image mega-menu cut conversion rate by 22.5% and orders by 22.7%, with a 99.5% probability that the control was better, on 10,957 mobile visitors in about a week.
Clean Commit reports Bayesian probabilities rather than p-values, but the lesson carries over: a one-sided "is it better?" frequentist test would only have said "not a winner."
Clean Commit's low-stock announcement bar on product pages is a similar case: -17.5% conversion rate and -11.3% revenue per visitor, with a 93.8% probability that the control was better, on 5,161 visitors in 6 days.
A scarcity message that most teams would expect to help went the other way.
Sometimes the expected direction is just wrong. FreestyleXtreme had long product descriptions above the images already live and ran the old page as the challenger.
The version without descriptions converted 31.38% better, at 96% confidence with over 7,000 visitors per version.
MarketingExperiments saw a three-step free trial sign-up on RegOnline's homepage cut completed leads by 24.5%.
If you only test the direction you hope for, results like these never get flagged as significant.
GrowthRock
Clean CommitClean Commit
MarketingExperimentsWhen each test is the wrong choice
One-tailed is the wrong choice when:
- You would act differently on a loss than on a tie. For example, you'd roll back a change that's already live, or investigate a bug. You need a test that can confirm the loss.
- You pick it after seeing the data. The UCLA FAQ warns that switching to one-tailed after a two-tailed test misses is "not appropriate, no matter how 'close' to significant the two-tailed test was." In practice, that move doubles your real false positive rate.
- You are reporting results to people who will read "not significant" as "safe." A one-sided test says nothing about harm.
- Your tool uses two-tailed math under the hood, as Optimizely's sequential engine does. Halving its p-value by hand breaks the error control it was built around.
Two-tailed is the wrong choice (or at least wasteful) when:
- A loss and a tie lead to the same decision: don't ship. Many new-feature tests work this way, and a one-tailed test gets you the answer with about 21-27% fewer visitors at common settings. That matters on low-traffic sites.
- You are running a non-inferiority test, such as checking that a cheaper or simpler page is not worse than the current one. The UCLA FAQ uses a drug version of this as its example of a fair one-tailed test.
- You read a direction out of a two-tailed result anyway. If you will only ever ship the variant when it wins, Georgiev's argument is that a one-sided test matches the claim you are actually making.
Whichever you choose, write it in the test hypothesis before launch.
Pair one-tailed tests with guardrail metrics so a harmful change still gets caught on revenue or other key metrics.
How to pick for your next test
Ask one question before launch: "If the variant is clearly worse, would I do anything different than if it were flat?"
If yes, use two-tailed. If no, a one-tailed test is legitimate and cheaper, as long as you lock it in up front.
Then keep the rest of your analysis consistent.
Use the same tail setting in your sample size plan and your significance read-out, and remember that a smaller p-value from a one-tailed test is not stronger evidence.
It is the same data answering a narrower question.
For the errors each choice trades off, see Type 1 and Type 2 errors, and for the bigger picture, our A/B testing guide.