A/B test QA is the check you run before and just after launch to confirm that every variation renders correctly, that visitors are split as planned, and that your goals and revenue are tracked the same way in every arm.
If QA is skipped, a broken variant or a missing conversion event can produce a "result" that is really a bug.
Most QA problems can be caught in under an hour with forced-variation links, a short device list and a sample ratio check.
This guide covers what quality assurance for an A/B test means, a step-by-step checklist, and real tests from our database where rendering bugs, device splits and split checks changed the outcome.
It fits between planning a test and running it.
What A/B test quality assurance covers
A/B test QA has a narrower goal than normal website QA.
You aren't only asking "does the page work?"
You're asking "would a difference between these arms be caused by the idea we're testing, and nothing else?"
That breaks down into four questions:
- Rendering: does each variation look and behave as designed on every device, browser and page template it will reach?
- Targeting and bucketing: are the right visitors entering the test, on the right pages, in the planned proportions, and do they stay in the same arm?
- Tracking: do the primary metric, revenue and guardrail metrics fire once per action, in every arm?
- Isolation: is the variant the only difference? Extra scripts, slower loads or a visible flicker all count as hidden changes.
Why it matters: bugs look like results
A bug in one arm doesn't throw an error in your testing tool.
It shows up as a lift or a drop, and that number gets reported like any other.
Our database has clear examples of how much a broken experience moves the numbers.
Clean Commit's mobile checkout rebuild reported orders doubled (+100%) despite 12.2% fewer visitors.
The agency itself notes the control had near-zero mobile conversions, so it was "more a broken-flow fix than a typical test."
If a bug like that sits in your variant instead of your control, you get a large loss that says nothing about your idea.
Surefoot's SMITH mobile cart redesign started from a cart with "rendering bugs on common devices" along with small tap targets.
The mobile-first cart got +8% visits to checkout on mobile at 95% significance.
Vasta's Canada Welding Supply header test replaced a third-party app header whose logo overlapped the search bar.
The theme-native header lifted conversion rate 17.02% and revenue per visitor 6.44% over a 14-day, 50/50 split.
Those were deliberate fixes. The QA lesson goes the other way: layout overlaps and device-specific rendering bugs are big enough to produce double-digit swings, so you have to rule them out in your own variants before you trust a result.
Device splits are a QA signal too
Some changes really do work on one device and fail on another.
Blend Commerce's two-image product cards for Stone Creek Coffee raised mobile conversion 20% but cut desktop conversion 12%.
Their simplified apparel homepage hero lifted desktop conversion 26% while mobile declined.
When you see a split like that, check the losing device for a rendering problem before you write it up as a behavior difference.
If the variant looks right on that device, the split is probably real, and the arm can ship where it won, as both of those tests did.
Vasta
Clean Commit
Surefoot
Blend CommerceQA checklist for an A/B test
Run the checklist in order. The first three stages happen before any real visitor sees the test; the last two happen after launch.
1. Before you build: write down what "correct" means
- Write the hypothesis, the primary metric and the guardrails, so you know which events must be checked.
- List the exact pages, templates and audiences the test should and should not reach.
- Record the planned traffic split (50/50, 85/15 and so on). You need this number later for the split check.
- Note any other tests running on the same pages, so you can check for interaction effects.
2. Preview each variation
Every major tool gives you a way to load a specific variation yourself without being bucketed at random.
For example, Optimizely Web Experimentation's optimizely_x parameter takes a list of variation IDs, overrides the visitor's natural bucket and disables tracking events, and optimizely_token lets anyone view draft or paused experiments for QA (Optimizely docs).
Convert offers preview URLs for checking how a variation looks, plus separate force variation URLs for final testing (Convert QA guide).
For each variation, including the control, check:
- Layout on your top devices and browsers by traffic share, at minimum one iPhone, one Android phone and one desktop browser.
- Every template the test touches: long and short product titles, sale and full-price items, out-of-stock products, products with and without reviews.
- Interactive pieces: menus, drawers, sliders, variant pickers, add-to-cart, quantity changes and form validation.
- Logged-in and logged-out states, and any currency or language versions in scope.
- Pop-ups, cookie banners and chat widgets that might cover the new element.
- Page speed and flicker. If the original content flashes before the change appears, fix it now. Running the test server-side is the most reliable way to stop it.
3. Test the full setup the way a visitor would
Previewing tells you the variation looks right. It doesn't tell you the experiment works.
For that, Convert recommends force variation URLs because they test "in the closest environment to what a visitor will experience," opened one at a time in fresh incognito windows, with a QA-only audience such as a query parameter (Convert QA guide).
- Restrict the live test to a QA audience (a query parameter, cookie or internal IP) and start it.
- Enter each arm in a clean incognito session. Confirm you land in the arm you forced and that you stay there after reloads and on later pages.
- Complete every goal: click the tracked buttons, submit the form, place a test order. Check that each conversion appears once in the test report and in your analytics tool, with the right revenue.
- Check that the pages outside the targeting rules stay untouched.
- If your site uses a consent banner, test with consent given and declined, so you know how each case is counted.
- Turn on the tool's debug output (Optimizely has optimizely_log for console logs) and look for errors in the browser console.
- Remove the QA audience before launch. If the experiment was ever live without a QA audience, Convert suggests cloning it rather than resetting it, because a reset doesn't clear the cookies of visitors who were already bucketed.
4. Right after launch: check the split
Within the first day or two, compare the visitor counts in each arm against the planned split.
A gap bigger than chance is a sample ratio mismatch (SRM), and it usually means something in the setup is dropping or adding visitors in one arm.
The standard check is a chi-square goodness-of-fit test:
chi-square = sum over arms of (observed - expected)^2 / expected
With two arms there is one degree of freedom.
Harness, for example, flags an SRM when the p-value is below 0.001 (Harness docs), a strict threshold that keeps false alarms rare.
| Case | Planned split | Observed | Expected | Chi-square | p-value | Verdict |
|---|---|---|---|---|---|---|
| Anatta buy-box test | 85/15 | 34,730 / 6,144 | 34,742.9 / 6,131.1 | 0.03 | 0.86 | No SRM |
| Made-up example | 50/50 | 10,000 / 9,500 | 9,750 / 9,750 | 12.8 | 0.0003 | SRM: stop and investigate |
The first row uses real counts from Anatta's tabbed buy-box test, which reported "SRM check passed" and a +2% AOV result ($102.63 to $104.74).
An uneven 85/15 split is fine as long as it matches the plan.
Dash of CX also reported "no SRM" alongside a 99% confidence loss in their Loop Earplugs checkout test (-3.04% conversion rate), which is the kind of note that makes a negative result easier to trust.
SRMs aren't rare. A KDD 2019 paper by Fabijan and colleagues reports that about 6% of Microsoft experiments had one, and LinkedIn found them in about 10% of triggered analyses (Fabijan et al., KDD 2019).
When the chi-square check is the wrong tool: it only tells you the split is off, not why.
It also can't catch a bug that hits both arms equally, or a tracking event that fires in one arm but not the other while traffic stays balanced. That's why stage 3 matters.
5. During the test: watch for breakage, not winners
- Check guardrail metrics (errors, page load time, refunds, support contacts) for sudden moves in one arm.
- Re-check the variant after any site deploy, theme update, app install or price change. A new release can break a client-side variant without anyone noticing.
- Look at the result by device and browser. A collapse on one browser is more often a bug than a preference.
- Don't stop the test early because a number looks good. That's an analysis problem, covered in common A/B testing mistakes.
When QA finds a problem
If the bug is in the variant and you catch it before launch, fix it and run the QA steps again.
If you catch it after launch, the data collected while it was broken is unreliable.
The clean option is to fix the bug and restart the test with fresh data, ideally as a new experiment so returning visitors are bucketed again.
If you see an SRM, don't analyze the result until you know the cause.
Common causes include redirects that lose visitors in one arm, bots, a variant that crashes before the tracking code loads, and targeting rules that treat the arms differently.
For a new tool or a new install, an A/A test is a useful extra check.
Two identical arms should show no difference and no SRM; if they do, the problem is in the setup, not in any idea you test later.
For the rest of the process, go back to the A/B testing guide.