The A/B testing mistakes that do the most damage are the ones that make a losing or neutral change look like a winner: stopping a test the moment it looks significant, running it on too few visitors, missing a broken traffic split, and judging it on the wrong metric. Each one gives you a real-looking number with nothing behind it. Below are 12 of them, grouped into seven areas, with real results from our database showing what each looks like in practice.
A fake win costs more than a loss. A loss gets thrown away. A fake win gets shipped, put into the forecast, and used to justify the next test.
Peeking: stopping the test when it looks good
1. Stopping as soon as the tool says "significant"
Most testing dashboards update in real time, so it is tempting to check daily and stop the first time the variant crosses 95% confidence. The trouble is that a classic fixed-sample test assumes you look at the result once, at the end.
Evan Miller's well-known essay How Not To Run an A/B Test shows that if you check after every observation and stop on the first significant result, a test set up for a 5% false positive rate can produce false positives 26.1% of the time. Early on, results swing a lot. If you stop on one of those swings, you lock in a false winner.
Ruben of Tailored Commerce saw exactly this when they added a doctor and expert endorsement badge to a product page. After week one the variant was up 17%. After four weeks it had settled at +1.66% CVR, which is flat. The week-one number was noise. Stopping early would have shipped a "17% winner" that did almost nothing.
Want results like this on your store?
Ruben at Tailored Commerce. Ruben runs CRO at Tailored Commerce for brands like Hears, Moov and FLÂNEUR.
2. Ending the test mid-week
Shoppers on a Tuesday lunch break behave differently from shoppers on a Sunday evening. If a test runs for nine days, two of those days count twice, and the result reflects that mix rather than a normal week. Run tests in whole weeks, and decide the length before launch. Our guide on how long to run an A/B test covers how to pick it.
Practitioners in our database are often open about results that are not finished yet. Kasper Dolk listed a product finder quiz on a collection page as "+€54,910 revenue lift, 89% confidence (in progress)". That label matters: a result that is still running is a progress report, not a verdict. If you want to check results along the way, use a method built for it, such as sequential testing or a Bayesian approach, and set the rules before you start.
Too small a sample
3. Not calculating sample size before launch
If you don't decide up front how many visitors the test needs, you have no way to know when it is done, which leads straight back to peeking. The sample you need depends on your baseline conversion rate and the smallest lift you care about detecting. Plug both into our A/B test sample size calculator before you build the variant, not after.
4. Hunting for small lifts on low traffic
A store with a few hundred orders a month cannot reliably detect a 2% lift. Small samples produce big swings, and the big swings are the ones that get reported. That is part of why a database of published wins skews toward large numbers.
Short windows deserve extra care. The Elevate A/B Testing team reported that moving the second product photo into first position took revenue per visitor from £3.32 to £3.60, worth +£785 in 10 days. That may well be a real gain, but ten days is less than two full weekly cycles. A result like that is worth re-running or watching after rollout before you count on it. If your traffic is limited, test bigger, bolder changes so the effect is large enough to measure. Our page on statistical significance explains what the confidence number does and does not tell you.
Sample ratio mismatch (SRM)
5. Not checking that traffic split the way you set it
A sample ratio mismatch happens when the visitor counts in each group differ from the split you configured by more than chance allows. You set 50/50 and got 50,000 visitors in control and 47,500 in the variant. Something is removing or adding visitors to one side, and those visitors are rarely random.
It is more common than most teams expect. In a KDD 2019 paper on diagnosing SRM, researchers from Microsoft, Booking.com and Outreach.io report that about 6% of experiments at Microsoft showed an SRM. The authors compare SRM to a fever: a symptom of many possible problems. Ignoring one, they warn, can make a bad change look good and get it shipped.
Common causes include:
- The variant's code crashes or redirects for some browsers, so those visitors never get counted.
- Bots or uptime monitors hit one version more than the other.
- Tracking fires on a page load that one version skips, such as a single-page app route.
- A variant was paused and restarted, or two tests overlap on the same visitors.
The check itself is simple: a chi-squared test comparing observed counts to the expected split. Convert's SRM guide says its built-in check flags a mismatch when p is below 0.01, and notes the check is off by default, so you have to switch it on in the project settings. Whatever tool you use, if a test shows an SRM, don't read the result. Find the cause, fix it, and restart.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
Novelty effect
6. Trusting the first week's lift
Returning visitors notice anything new. A fresh badge, a moving element or a new layout can pull clicks simply because it is different, and that pull fades as people get used to it. The reverse also happens: regulars can resist a change at first and then adjust.
The authority badge test above is a good example of a first-week lift that faded. One way to check: compare the lift for new visitors against returning visitors, and look at the trend by week. A lift that shrinks every week is a warning sign, even if the overall result still shows significance.
7. Running a test through a sale or holiday
Black Friday shoppers, sale traffic and gift buyers behave differently from your usual customers. A change that wins during a promotion may not win in March. If a test has to overlap with a sale, either extend it well past the sale or treat the result as specific to promo traffic.
Wrong metric
8. Calling a winner on conversion rate alone
Conversion rate counts orders, not money. A discount, a cheaper default option or a free gift can raise conversion rate while revenue per visitor or profit falls. That is why most practitioners in our database report revenue per visitor (RPV) or profit per visitor as the main metric. Our guide to A/B testing metrics covers how to pick one.
Offers are where this bites hardest. Milos changed a 2-pack offer to "Buy 2 Get 1 Free" at the same price, and RPV fell 52.38%, a loss estimated at $104,679 a month. A test judged on units moved or add-to-cart rate could easily have called that a win.
9. Ignoring what happens after the order
Some changes trade revenue today for revenue later. Dave Diederen introduced subscriptions with the option pre-checked by default. Upfront RPV dropped 9.17%, while subscription share rose 1,295%. Judged on first-order RPV alone, that is a loss. Judged on lifetime value, it may be the best test of the year. Decide which metric settles the call before the test starts, and write it down.
Profit tells a different story again. Kasper Dolk's benefits-first hero lifted conversion rate 14.41% and profit per visitor 21.97%. Dave Diederen raised a free shipping threshold to $165 and reported +6.27% profit per visitor, a change that conversion rate alone might have flagged as risky.
Segment cherry-picking
10. Digging through segments until something wins
When a test comes back flat, it is tempting to slice it by device, country, traffic source and new vs. returning until one segment shows a lift. Cut the data enough ways and something will look significant by chance alone. A segment "win" found after the fact is a new hypothesis, not a result. Run it as its own test, with that segment set as the target before launch.
11. Assuming one store's result applies everywhere
The reverse mistake is treating a single result as a rule. Countdown timers are the clearest example in our database. Dave Diederen added a countdown timer to checkout and saw +3.16% RPV. A second checkout countdown test lost 7.1% RPV. A countdown timer for a European market lost 35.61% RPV.
Ruben ran the idea across several brands and got about +10% CVR on a low-priced brand, flat results on a high-priced brand, and a 9% CVR loss on another. Same idea, opposite outcomes, depending on audience, price point and market. That is why the tests in our database are evidence for what to try on your store, not a promise of what will happen. For more on how often good-looking ideas fail, see our research on losing A/B tests.
Want results like this on your store?
Dave Diederen at Syntra Labs. Dave builds and tests Shopify stores at Syntra Labs, and shares most results in public.
Flicker and tracking bugs
12. Shipping a test that flickers or tracks wrong
Client-side testing tools change the page in the browser after it starts loading. If the script loads late, visitors briefly see the original page before the variant appears. Convert's glossary entry on the flicker effect describes it as a flash of original content, and notes it can confuse visitors, push them to bounce and skew results. In practice, the variant gets judged partly on a glitch rather than on the idea.
Tracking bugs are quieter but just as harmful. A purchase event that fires twice in one version, a variant that breaks on one browser, or a revenue goal pointing at the wrong page can all create a "win" out of nothing. Before you launch, run this quick QA:
- Load each variant on mobile and desktop, with a cleared cache, and watch for flicker.
- Place a test order in each variant and confirm it is recorded once, with the right revenue.
- Check the visitor split after the first day for signs of an SRM.
- Compare the testing tool's order count with your store's own order report at the end.
If you are choosing a tool, flicker handling and SRM alerts are worth checking in the docs before you buy. Our Convert review looks at how one vendor handles both.
A short pre-launch checklist
- Write the hypothesis, the main metric and the sample size before you build anything.
- Run in full weeks and don't stop early unless your method allows it.
- Check the traffic split for SRM early and again at the end.
- Look at the weekly trend for novelty effects.
- Judge on RPV or profit, not conversion rate alone.
- Treat segment wins as new hypotheses.
- QA for flicker and tracking before a single visitor sees the test.
Questions
What are the most common A/B testing mistakes?
The most common ones are stopping a test as soon as it looks significant (peeking), running it on too few visitors, missing a sample ratio mismatch, trusting a first-week novelty lift, judging on conversion rate instead of revenue or profit per visitor, cherry-picking segments after the fact, and shipping tests with flicker or broken tracking.
What is a sample ratio mismatch?
A sample ratio mismatch (SRM) is when the number of visitors in each test group differs from the split you configured by more than chance allows, for example a 50/50 test that ends up 50,000 vs. 47,500. It points to a bug such as a crashing variant, bot traffic or broken tracking. A KDD 2019 paper reports that about 6% of experiments at Microsoft showed an SRM. If you see one, fix the cause and restart the test.
How do you check for sample ratio mismatch?
Run a chi-squared goodness-of-fit test comparing the observed visitor counts in each group to the expected split. A very low p-value means the mismatch is unlikely to be chance. Convert's built-in SRM check uses a threshold of p below 0.01, but it has to be turned on in settings.
Why is peeking at A/B test results a mistake?
A standard fixed-sample test assumes you look at the result once. Checking repeatedly and stopping at the first significant reading inflates false positives: Evan Miller shows that checking after every observation can push a 5% false positive rate to 26.1%. Set the sample size in advance, or use a sequential or Bayesian method designed for interim looks.
Can a flat A/B test still have a winning segment?
It can look like it, but a segment win found after the test is usually chance from slicing the data many ways. Treat it as a new hypothesis and run a separate test targeted at that segment before acting on it.