The A/B testing interview questions that come up most are about sample size and power, peeking, sample ratio mismatch, statistical significance, and what you do when a result looks too good.
Strong answers explain the method, give a number, and say when the method breaks.
Below are the common experimentation and CRO interview questions with model answers, each backed by a real test from our database.
Interviewers for data science, product analytics and CRO roles rarely want a textbook definition.
They want to see that you can spot a broken test, size one before it starts, and explain a result to someone who doesn't know statistics.
Experimentation interview questions
These are the statistics and design questions. Product analytics and data science interviews lean on them most heavily.
1. How do you decide how many visitors a test needs?
Start from four inputs: the baseline conversion rate, the smallest effect worth detecting (the minimum detectable effect), the significance level, and the power you want.
For a conversion rate, the usual normal-approximation formula per variant is:
n = (z_alpha/2 + z_beta)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2
Worked example: a 3.0% baseline, and you want to detect a 10% relative lift to 3.3%, at 95% confidence (two-sided, z = 1.96) and 80% power (z = 0.84).
That gives (2.80)^2 x (0.0291 + 0.0319) / 0.003^2, or about 53,200 visitors per variant.
Halve the lift you want to detect and the sample roughly quadruples.
A good answer adds when the formula is wrong.
It assumes a roughly normal sampling distribution. Ron Kohavi and colleagues warn that many online metrics, such as revenue, are skewed, and need larger samples than the formula suggests (Kohavi et al., KDD 2014).
See our page on statistical power for the full walk-through.
2. What is statistical power, and why does low power matter even when a test "wins"?
Power is the chance a test detects a real effect of a given size.
Most teams aim for 80%. The part interviewers listen for: low-powered tests that do reach significance tend to overstate the effect. This is the winner's curse.
A concrete case: in a Conversion Fanatics test that expanded all product options on mobile, the variant led by more than 40% early on, then faded to about -4% conversions by the end.
Early numbers from a small sample are noisy, and the big ones are the least trustworthy.
3. Why shouldn't you stop a test the moment it hits significance?
Because checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate far above 5%.
Evan Miller's simulation shows that peeking after every observation and stopping at the first significant result pushes the false positive rate to 26.1% (Evan Miller).
The fix is either to fix the sample size in advance and look once, or to use a method built for continuous monitoring, such as the always-valid p-values described by Johari, Pekelis and Walsh.
Our sequential testing page covers the trade-offs. Sequential methods are the wrong choice when you have a fixed deadline and enough traffic anyway: they cost some power for the right to look early.
4. What is a sample ratio mismatch, and how do you check for one?
An SRM means the observed traffic split differs from the split you configured by more than chance allows.
It usually signals a bug in assignment, redirects or tracking, and it makes the result untrustworthy.
A KDD 2019 paper found SRMs in about 6% of Microsoft experiments (Fabijan et al., 2019).
The check is a chi-square goodness-of-fit test. Worked example from Anatta's tabbed buy box test, which used an 85/15 split and reported 34,730 control and 6,144 variant visitors.
Expected counts are 34,742.9 and 6,131.1. The chi-square statistic is about 0.03 with 1 degree of freedom, a p-value of about 0.86 (our calculation), so there's no mismatch.
Anatta reported the SRM check passed. Many tools flag an SRM at p < 0.001. More on this in sample ratio mismatch.
5. One-tailed or two-tailed test?
A one-tailed test only looks for an effect in one direction, so it needs less traffic to reach significance in that direction.
The catch is that you have to choose before the test, and you give up the ability to call a loss significant.
The UCLA statistics FAQ calls switching after seeing the data "not appropriate".
The choice changes real conclusions. GrowthRock's mobile homepage link bar test showed +5% orders with 93% significance one-tailed, across 80,000+ visitors and 2,300 conversions per variation.
It was not significant under Optimizely's two-tailed stats, so it counts as inconclusive. See one-tailed vs two-tailed tests.
6. A test shows a huge lift. What do you do?
Suspect a bug first. Kohavi's team calls this Twyman's law: "Any figure that looks interesting or different is usually wrong" (Kohavi et al., KDD 2014).
Check the SRM, the tracking, the conversion counts, and whether the control behaved like it normally does.
A MarketingExperiments test on a tech supplier's product page is the textbook case.
One treatment showed +445.6% at 99% confidence, but there were only 2-10 conversions per day and the control had six straight days with no conversions.
A rerun with a cloned control showed a 30% gap between identical pages, so the lift wasn't trusted.
7. How would you confirm a result is real?
Rerun it, or hold back a group after launch.
Voices.com removed featured talent profiles from its homepage, saw +33% sign-ups in a 10-day 50/50 test, then reran it and got +28%.
A replication that lands near the first result is far stronger evidence than one p-value.
A smaller second number is normal: first results that cleared a significance bar tend to be inflated.
8. Results differ by device. What does that tell you?
That the treatment interacts with the segment, and the pooled number hides it.
Blend Commerce's two-image product card test for Stone Creek Coffee found +20% conversion rate on mobile and -12% on desktop, so the change shipped on mobile only.
A good answer also warns against slicing into many segments after the fact and treating each as a fresh test: that's a multiple comparisons problem. See interaction effects.
Blend Commerce
GrowthRock
Conversion Fanatics
MarketingExperimentsCRO interview questions
CRO and growth interviews spend less time on formulas and more on process, prioritization and judgment.
These are the questions that separate people who have run tests from people who have read about them.
9. What share of your tests do you expect to win?
Fewer than most people think, and an honest answer earns trust.
Kohavi and colleagues reported that at Microsoft only about one third of ideas improved the metric they were designed to improve (Kohavi et al., 2009).
Optimizely's benchmark of its customers' experiments puts wins on the primary metric at about 12% (Optimizely).
Our own data shows the gap between what gets published and what happens.
Most results in our database are wins, because people share winners.
Clean Commit publishes its whole library: of its 173 tests in our database, 16 won, 35 lost and 122 were flat, about a 9% win rate.
10. Walk me through how you'd find what to test.
Interviewers want a research step before any idea.
A solid answer: start with analytics to find where people drop off, then use session recordings, surveys and user testing to learn why, then write a hypothesis that names the problem, the change and the metric you expect to move.
Mention how you'd rank the list, for example by expected impact, confidence and effort.
Ground it in evidence. Before proposing a change, look at how similar tests went for other sites in the A/B test database.
A pattern across several tests is a stronger prior than one blog post.
11. Which metric would you use as the primary metric?
The one closest to revenue that still has enough events to reach a conclusion in reasonable time.
For an ecommerce store that is often orders or revenue per visitor.
Click-through to the next step is easier to move but can rise while sales fall.
Add guardrail metrics that must not get worse, such as refunds, page speed or unsubscribe rate.
Then say what you'd do if the primary metric wins and a guardrail loses: usually, not ship until you understand why.
12. Tell me about a test that lost. What did you learn?
This tests honesty as much as skill. Pick a real loss, explain the hypothesis, what happened, and what you changed in your next test.
Teams that learn from losses usually find a better follow-up test than teams that only chase wins.
13. Your site has low traffic. How do you still test?
Test bigger changes, so the effect you need to detect is larger.
Pick a metric higher in the funnel with more events, run fewer variants, and accept longer tests.
Where a test truly can't reach a useful sample, say so and use qualitative research or a before-and-after read with clear caveats instead of a fake A/B test.
14. How do you explain a result to a non-technical stakeholder?
Give the range, not just the point estimate: "orders rose about 5%, and the plausible range runs from a small loss to about a 10% gain" is more honest than "we won by 5%".
Say what you recommend, how sure you are, and what it's worth in money if it holds.
How to prepare
Practice saying each answer out loud with a number in it.
Work through one sample size calculation and one SRM check by hand, and have two real tests of your own ready: one win and one loss.
If you're still deciding on the path, our guide to becoming a CRO specialist covers the skills these interviews probe, and the CRO guide covers the process questions in depth.