SEO A/B testing (also called SEO split testing) measures how a change to your pages affects organic search traffic.
You split a group of similar pages, not visitors, into control and variant sets, change only the variant pages, and compare their organic clicks against a forecast of what they would have earned without the change.
That makes it a different experiment from the conversion tests in our database.
A normal A/B test asks "do people who see version B buy more?"
An SEO test asks "does Google send more people to pages that use version B?"
This page covers how to run the second kind, and how to run the first kind without harming your rankings.
How SEO split testing works
In a user-split test, each visitor is randomly assigned a version.
That can't work for search, because there is effectively one visitor that matters: the search engine's crawler.
It sees one version of each URL and ranks that URL accordingly.
So SEO tests randomize at the page level.
Each page has one version for everyone, people and crawlers alike.
The unit of analysis is a page's daily organic sessions or clicks, not a visitor's conversion.
Step by step
- Pick a template with many similar pages. Product pages, category pages, location pages, recipe pages or listings. You need hundreds of pages, ideally more.
- Bucket the pages into control and variant. Assign them randomly, or use stratified ("smart") bucketing so both groups have similar traffic levels and volatility before the test starts.
- Change only the variant pages. Typical changes: title tags, meta descriptions, headings, structured data, internal links, intro copy, or content blocks such as FAQs.
- Measure organic traffic to both groups over time. Daily organic sessions from analytics, or clicks from Search Console.
- Compare the variant against a counterfactual. Use the control group to estimate what the variant pages would have done without the change, then measure the gap.
Why you can't just compare the two groups
Pages are far less uniform than visitors. One page might get 40 visits a day and the next 4,000, so two groups of pages rarely match exactly, even after random assignment.
SearchPilot, which sells SEO testing software, describes this as the core difference: pages vary more than users, and there are far fewer of them to split.
The fix is to compare each group with its own past, using the control group to soak up shared noise like seasonality, Google algorithm updates and competitor moves.
Etsy's engineering team ran title tag experiments this way: pages were randomly assigned to groups with different title tag phrasings, and because small pre-test differences remained between groups, they chose Google's CausalImpact package over a simple difference-in-differences comparison, building a synthetic control group to estimate the effect.
CausalImpact comes from a 2015 paper by Brodersen and colleagues at Google that uses Bayesian structural time-series models to predict the counterfactual: the traffic the changed pages would have received with no intervention.
SearchPilot started with that model and moved to a neural network in 2019, saying the change let it detect uplifts half the size of its previous smallest.
Worked example: difference-in-differences
Say a store has 2,000 product pages and splits them into two groups of 1,000. The variant pages get a new title tag pattern that adds the price range.
Here is weekly organic sessions, averaged over 8 weeks before and 4 weeks after the change.
| Before (sessions/week) | After (sessions/week) | Change | |
|---|---|---|---|
| Control pages | 52,000 | 49,400 | -5.0% |
| Variant pages | 50,000 | 50,500 | +1.0% |
The control pages fell 5%, probably from seasonality or an algorithm shift, since nothing about them changed.
So the expected variant traffic without the change is:
expected variant = variant before × (control after ÷ control before) = 50,000 × (49,400 ÷ 52,000) = 50,000 × 0.95 = 47,500
estimated lift = (actual ÷ expected) − 1 = (50,500 ÷ 47,500) − 1 = +6.3%
Notice how the two shortcuts get it wrong.
A before/after check on the variant alone says +1.0%, because it ignores the sitewide drop.
A straight comparison of the two groups after the change says +2.2% (50,500 vs 49,400), because the groups weren't equal to begin with.
Neither number is the effect of the title change.
This example gives a point estimate only. To know whether +6.3% is real or noise, you need an interval, which means modeling the daily variation across pages, for example with CausalImpact or by re-running the analysis on many random reshuffles of the page groups (the bootstrap approach SearchPilot says it uses).
If the interval includes zero, treat the result as inconclusive, the same way you would read a significance test on a conversion experiment.
How much traffic you need
There is no universal minimum, because it depends on how stable your pages' traffic is and how small an effect you want to detect (your minimum detectable effect).
For reference, SearchPilot said in 2019 that its old rule of thumb was at least 30,000 organic sessions a month to the group of pages being tested, and that its newer model had run valid tests on sections with under 10% of that.
Those figures are for its own model, so treat them as a rough guide only.
Etsy reported that effects showed up within a few days of the start date, and that rolling the winning title out to all groups produced similar lifts in the others, which is a useful extra check.
Still, no effect can appear until Google has recrawled the changed pages, so give the test at least that long plus enough days for the interval to narrow.
When SEO split testing is the wrong choice
- You have only a handful of pages. You can't split a homepage or 12 service pages into meaningful groups. A single-page change can only be judged before/after, which can't separate your change from seasonality or algorithm updates.
- The change is sitewide. A new navigation, a site migration or a speed overhaul touches every page, so there is no control group left. Use a dated before/after analysis and be honest that it shows correlation.
- Traffic is low or spiky. If daily sessions per page swing wildly, the model's interval will be too wide to detect a realistic effect. See testing with low traffic for alternatives.
- You're testing title tags that Google may not show. Google says it can build title links from headings, og:title, anchor text and other page content, not just the title element. If Google rewrites many of your variant titles, you are testing less than you think. Check what actually appears in results.
- You care about conversions, not visits. An SEO test tells you whether a change brings more people in. Whether those people buy is a separate question, answered by a user-split test.
SEO testing for conversion tests: what to check and what our data shows
"SEO testing" also covers the other direction: making sure your conversion experiments don't hurt your search performance.
Google's guidance on running website tests is short and clear.
- Don't cloak. Google says not to show one set of URLs to Googlebot and a different set to humans, which breaks its spam policies. Don't exclude the crawler from your test or always serve it the control.
- Use rel=canonical on alternate test URLs. If a split URL test sends visitors to a separate variant URL, point the variant's canonical tag at the original. Google prefers this to noindex.
- Use 302 redirects, not 301. A temporary redirect tells search engines to keep the original URL in their index.
- End tests promptly. Google asks you to ship the winner and remove the test setup as soon as possible, and warns that a test running much longer than necessary, especially one serving a variant to most users, may be seen as an attempt to deceive.
How you deliver the test matters too. Client-side tools change the page in the browser after it loads, which can cause flicker and slower rendering.
Server-side delivery avoids that; see server-side vs client-side testing for the trade-offs.
Organic landing pages convert differently
Pages built to rank are often written for crawlers first, and conversion tests on them have found big room for improvement.
HDFC ERGO's agency, Performics India, tested two organic insurance pages on mobile.
A stronger hero and a "Free" quote CTA on the car insurance page lifted lead conversion rate by 38%, and putting the full form above the fold on the health insurance page lifted it by 25%.
Both tests ran only 9 days and changed several things at once, so they don't tell you which change did the work.
MarketingExperiments reported a far larger result on a SaaS SEO landing page for RegOnline: a radical redesign with the top navigation removed and a single CTA raised sign-ups by 548%.
It was a full redesign, so it shows the page had room to improve, not which element mattered.
A smaller, cleaner example: Visually tested long SEO product pages for Cornbread Hemp that stacked several products one after another.
Adding a collection-style overview at the top linking to each product lifted per-session value by 6.8%, while leaving the long content that helps the pages rank in place.
SEO copy and conversion copy can coexist
You don't always have to choose between ranking and converting.
Traffic4U ran a 2x2x2 multivariate test on Hyundai Netherlands car model pages.
The best combination used SEO-oriented text, extra CTA buttons and a large car photo, and it increased test-drive and brochure requests by 62%.
Content often added for SEO, such as FAQs and breadcrumbs, doesn't reliably move conversions either way.
Clean Commit tested review-based FAQs on collection pages (+8.4% conversion rate, 83.4% probability to beat control, not significant) and breadcrumbs on collection pages (+1.8% conversion rate, inconclusive).
If you add these for search, a conversion test is a cheap way to confirm they don't cost you sales.
Organic traffic dilutes conversion results
Search visitors often arrive with research intent and no plan to buy.
Patrick McKenzie tested a new logo on Bingo Card Creator with about 8,500 people per variation: trial conversion fell from 11.05% to 9.92%.
Patrick McKenzie noted that sitewide conversion included lots of SEO traffic that need not sign up, which pulls every variation's rate toward zero and makes effects harder to detect.
If organic visitors are a large share of a test, segment results by channel and pick a test length that accounts for the diluted baseline.
Visually
MarketingExperimentsThe short version: use page-split SEO tests to learn what brings visitors in, user-split A/B tests to learn what makes them convert, and Google's canonical, 302 and no-cloaking rules to keep the second from undermining the first.