A/A testing means running an experiment where both groups see the exact same page, so any difference you measure is caused by chance or by a fault in your setup.
It doesn't tell you what converts better. It tells you whether your testing tool splits traffic evenly, tracks conversions correctly and reports significance at the rate it claims.
In a healthy setup, only about 5% of A/A tests should come out "significant" at the 95% level.
That last number is the whole point. If your tool calls a winner between two identical pages much more often than that, the problem is in your setup, not your pages, and every A/B test you run on it carries the same flaw.
Below we explain what an A/A test is, how to run one, how to read the result, and when it's a waste of traffic.
Along the way we point to real tests from our database where a big-looking lift turned out to be noise, which is exactly the kind of mistake A/A testing teaches you to spot.
What an AA test is and what it catches
An AA test (also written A/A test) splits visitors at random into two groups, just like an A/B test.
The difference is that the "variant" is a copy of the control.
Same design, same copy, same code. In the language of control and variant, you have two controls.
Because nothing differs, the null hypothesis is true by design.
Microsoft's Experimentation Platform team puts it this way: with A/A tests, you don't have to assume the null hypothesis is true, you know it is (Microsoft Research).
That makes an A/A test the one experiment where you know the right answer before you start.
Optimizely's documentation describes it as a "data reliability and quality assurance procedure" and says most A/A results should come back statistically inconclusive (Optimizely docs).
The book Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu gives it a full chapter, "The A/A Test" (Cambridge University Press).
What a failed A/A test usually points to
- Uneven traffic split. You asked for 50/50 and got 52/48. This is a sample ratio mismatch, and it means something other than chance decides who lands in which group. Bot filtering, redirects and tracking that fires only on one arm are common causes.
- Broken conversion tracking. A goal that fires twice on one arm, or misses orders from a payment method, shows up as a "difference" between identical pages.
- Wrong variance math. If the tool treats every pageview as independent when the unit is really the user, its confidence intervals come out too narrow and it calls winners too often. Microsoft's team found that across a dozen products, 10-15% of metrics failed their simulated A/A checks, with some products as high as 30% (Microsoft Research).
- Outliers and rare events. One customer placing a huge order, or a metric that only a handful of users ever trigger, can swing a revenue or rare-event metric and produce lopsided p-values (Microsoft Research).
- Script side effects. The testing snippet itself can slow pages or cause the flicker effect. An A/A test won't isolate this unless you compare against traffic outside the test, but it does confirm the snippet doesn't treat the two arms differently.
Why a "winner" between identical pages isn't a surprise
A 95% significance threshold means you accept a 5% false positive rate.
Run 20 A/A tests and you should expect about one to come out "significant".
The chance that at least one of the 20 does is 1 - 0.95^20, or about 64%.
So one significant A/A test proves nothing on its own.
What matters is the rate over many tests, or over many metrics in one test, which is also why multiple comparisons matter.
Microsoft's team describes the healthy pattern precisely: across many A/A tests, p-values should be spread evenly between 0 and 1. About 5% should fall below 0.05, about 20% below 0.2, and so on (Microsoft Research).
If p-values pile up near zero, your tool is overconfident.
If they bunch up in a few values, the metric is too rare or too lumpy to test reliably.
The same noise shows up in real A/B tests
You don't need an A/A test to see how much random noise moves a conversion rate.
Our database has plenty of real A/B tests where a large lift wasn't real:
- Ruben's team tested an authority badge on a product page. After one week it showed +17% conversion rate with a 99.6% probability to beat control. They kept it running, and after four weeks the lift was down to +1.66%, which they judged to be noise.
- Clean Commit's test of a shorter mobile product gallery showed +24% conversion, not significant.
- Clean Commit's test of sending ad traffic to the homepage showed +103.9% conversion on a tiny sample, and is logged as flat.
- Blend Commerce's mobile sticky add-to-cart bar showed +10% conversion, not significant.
None of these were A/A tests. But they show the size of swings that chance alone can produce at typical ecommerce traffic levels, which is the lesson an A/A test teaches with your own data.
Clean CommitClean Commit
Blend CommerceClean Commit
@RubendgrfHow to run an A/A test and read the result
Set it up exactly as you would a real A/B test, with the same tool, targeting, goals and planned length.
The only difference is that the variant has no changes.
- Pick the page and metrics you test most. Usually that's a product page or landing page, with conversion rate and revenue per visitor as goals.
- Fix the sample size and duration in advance. Use the sample size calculator as if you were testing a real change, and run full weeks. See how long to run an A/B test.
- Don't stop early. Evan Miller showed that checking results after every new batch of data and stopping at the first p < 0.05 pushes the real false positive rate on a true null to 26.1%, not 5% (Evan Miller). An A/A test is a good place to see this happen.
- Check the split first. Run an SRM check on visitor counts before looking at any conversion numbers.
- Then check the metrics. Expect no significant difference on your primary metric, and only an occasional one across secondary metrics.
Worked example: reading an A/A result
Say your store converts at 3.0% and each arm gets 10,000 visitors.
The standard error of the difference between two conversion rates is:
SE = sqrt(p(1-p)/n1 + p(1-p)/n2) = sqrt(0.03 x 0.97 / 10,000 x 2) = 0.0024
The 95% margin is 1.96 x 0.0024, about 0.47 percentage points.
So two identical pages can land anywhere from roughly 2.8% vs 3.2% without anything being wrong.
In relative terms, that's a gap of up to about 16% between arms that show the same page. See statistical significance for the full method.
Now the split. You expected 50/50 and got 10,000 vs 10,400 visitors.
A chi-square test against the expected 10,200 per arm gives:
chi-square = (10,000 - 10,200)^2 / 10,200 + (10,400 - 10,200)^2 / 10,200 = 7.84
With one degree of freedom, that's a p-value of about 0.005. A 2% gap in traffic sounds small, but it's very unlikely to happen by chance, so you'd investigate before trusting any test on this setup.
Our SRM checker runs this math for you.
Microsoft reports that about 6% of its A/B tests have an SRM (Microsoft Research), so this isn't rare.
| What you see in the A/A test | What it likely means | What to do |
|---|---|---|
| Split matches, no significant difference | Setup is working | Start A/B testing |
| Split matches, one metric significant | Probably chance (about 1 in 20) | Rerun or check over more tests before acting |
| Split is off (SRM) | Assignment or tracking fault | Fix before running real tests |
| Many A/A tests or metrics significant | Tool is overconfident or tracking is broken | Check the analysis unit, goal firing and outliers |
| Revenue metric swings, conversion rate doesn't | Outlier orders | Cap or trim extreme orders in analysis |
When an A/A test is the wrong choice
An A/A test costs the same traffic as a real test and can't produce a win.
On a low-traffic site, a four-week A/A test is four weeks of no learning, and with small samples it may be too noisy to catch anything but gross errors.
It's also the wrong tool for checking that one specific variant renders correctly.
That's what test QA is for. And a single passing A/A test doesn't prove your tool is calibrated. It only rules out large faults.
Run one when you install a new testing tool, change how traffic is assigned, add new conversion tracking, or see results that look too good.
If you have historical data, the Microsoft approach of simulating many A/A tests by re-splitting past users at random checks calibration without spending live traffic (Microsoft Research).
Teams with high traffic can also keep an A/A arm running in the background as an ongoing health check.
Once your setup passes, the real work starts.
Browse real test results for ideas worth testing.