A/B testing (also called split testing) means showing two versions of a page to random groups of visitors at the same time, then keeping the version that earns more. For an online store, that usually means testing a product page, cart, or checkout change against the current version and measuring conversion rate or revenue per visitor. It replaces opinions with evidence: a change ships only if real shoppers prove it works.
This guide covers what A/B testing means, how a test runs, what stores test, how much traffic you need, how to read results, and which tools to use. Every example comes from a real test in the LiftArchive test database, credited to the practitioner who ran it.
What is A/B testing? A/B vs split vs multivariate
In an A/B test, version A is the control (what you have now) and version B is the variant (the change). Visitors are randomly assigned to one version, and each person keeps seeing the same version for the whole test.
Because both groups shop during the same days, outside factors like ad spend, holidays, and email sends hit both sides equally. That is what makes A/B testing more reliable than a before-and-after comparison, where a sale or a viral post can make a bad change look good.
How online stores use A/B testing
Stores use A/B tests to answer practical questions: does a guarantee near the buy button help, is free shipping worth the margin, does a shorter product page sell more? Each answer is measured in orders and revenue, not clicks or opinions.
Over time, a steady testing program builds a record of what works for one specific audience. That record is worth more than any single win, because it makes every future design and offer decision faster.
Split testing, A/B/n, and split URL tests
"Split testing" and "A/B split testing" usually mean the same thing as A/B testing. The names come from splitting traffic between versions.
An A/B/n test adds more variants, such as three headlines against one control. Each extra variant needs its own share of traffic, so the test takes longer.
Some tools use "split URL testing" for a narrower case: the two versions live at different URLs, such as two separate landing pages, and visitors are redirected between them. That is handy when the variant is a full redesign that is easier to build as a new page.
Multivariate testing
Multivariate testing changes several elements at once (say, two headlines and two images) and tests every combination to see which elements matter and how they interact. Four combinations need roughly four times the traffic of a simple A/B test, so most stores rarely have enough visitors for it. Our guide to multivariate testing covers when it is worth it.
| Method | What changes | Traffic needed | Best for |
|---|---|---|---|
| A/B test | One version against the control | Lowest | Most ecommerce tests |
| A/B/n test | Several variants against one control | Grows with each variant | Comparing 2-3 offers or headlines |
| Split URL test | Whole page at a different URL | Same as A/B | Redesigns and new landing pages |
| Multivariate test | Several elements, every combination | Highest | High-traffic stores tuning one page |
Key terms
- Control: the current version, used as the baseline.
- Variant (or treatment): the changed version.
- Conversion rate (CVR): the share of visitors who place an order.
- Revenue per visitor (RPV): total revenue divided by visitors. It captures both conversion rate and order value, so it is often the better primary metric for stores.
- A/A test: two identical versions run against each other. If your tool reports a big "winner" between identical pages, something is wrong with your setup, not your store.
How an A/B test works
Every solid test follows the same loop. The full walkthrough is in how to run an A/B test, but here is the short version.
- Find a problem. Use analytics, heatmaps, session recordings, and customer feedback to find where shoppers drop off or hesitate.
- Write a hypothesis. For example: "Because mobile shoppers can't see our guarantee, adding it under the add-to-cart button will raise conversion rate." A clear hypothesis tells you what to build and what to measure.
- Pick one primary metric. For most stores this is conversion rate or revenue per visitor. Decide it before the test starts.
- Work out the sample size. Use a sample size calculator to see how many visitors each version needs and how many weeks that takes.
- Build the variant and QA it. Check it on mobile and desktop, in the main browsers, and through a real test order.
- Run the test for full weeks. Do not stop it early because one side is ahead.
- Read the result and decide. Ship the winner, drop the loser, and write down what you learned either way.
Under the hood, the testing tool assigns each visitor to a group, usually with a cookie or a server-side rule, and records what they do. Client-side tools change the page in the browser with a script. Server-side tools send each visitor a different version from the server, which avoids the "flicker" of the original page loading first.
Why the metric choice matters
The metric you pick can flip the verdict. When Dave Diederen pre-checked a subscription option by default, revenue per visitor fell 9.17% up front, while subscription share rose 1,295%. Judged on RPV alone it lost. Judged on long-term customer value it may have won.
The same goes for conversion rate. An offer that gets more people to buy cheaper bundles can raise conversion rate and still lower revenue. Track RPV and average order value next to conversion rate, even when one of them is your primary metric.
Write down every result
Keep a simple log of each test: the hypothesis, the page, the dates, the sample size, the result, and the confidence level. After a dozen tests, the log tells you which kinds of changes work for your customers. That is how our database started: practitioners sharing what they learned so others don't repeat the same tests blind.
What to test: ideas from real ecommerce tests
The best test ideas fix a specific reason shoppers don't buy: doubt about quality, unclear value, surprise costs, or friction. Here is what our database shows practitioners testing, and what happened.
Product page copy and trust
Dave Diederen saw +10.47% conversion rate after rewriting a product title as a benefit-led headline, turning a plain product name into the outcome the customer wants. Kasper Dolk measured a +$141,673 revenue lift at 100% confidence from a benefit-led buy box proposition on the product page.
Risk reducers also show up often. Oliver Kenyon reported +€63,000/month after adding 60-day guarantee and easy returns blocks under Add to Cart. Dave Diederen saw +7.48% conversion rate from putting a "60-Day Money-Back Guarantee" inside the add-to-cart button itself.
Cart and checkout
The last steps before payment are where small doubts cost sales. Oliver Kenyon saw +23.8% revenue per visitor from a cart drawer change that added a dispatch, returns, and payment block under the checkout button. Dave Diederen saw +9.90% conversion rate after adding three USPs (guarantee, orders shipped, delivery speed) below the Pay Now button.
Pricing, offers, and shipping
Offers are high-stakes tests because they can win or lose big. Dave Diederen found that a $9.49 shipping fee beat the control by +43.81% RPV. In the other direction, Milos saw RPV drop 52.38% (an estimated -$104,679/month) after changing a 2-pack offer to "Buy 2 Get 1 Free" at the same price.
Price and offer tests touch margin, not just conversion, so check profit per visitor before you call a winner.
Changes that "should" work but didn't
Some popular advice fails in testing. Dave Diederen removed an email popup expecting a cleaner experience and lost 20.51% RPV. Skipping the cart and sending shoppers straight to checkout cost 10.23% conversion rate in another of Dave's tests. These are exactly the changes a store would ship without testing.
Browse all 285 results by page type and element in the test database, or get a ranked list of ideas on the test ideas page. Start with the pages that get the most traffic and the biggest drop-off, since that is where a winning test pays back fastest.
Want results like this on your store?
Dave Diederen at Syntra Labs. Dave builds and tests Shopify stores at Syntra Labs, and shares most results in public.
How long to run an A/B test and how much traffic you need
Run a test until it reaches the sample size you calculated before launch, and for at least one or two full weeks so every weekday is covered. Shopping behavior on a Monday morning differs from a Saturday night, and a test that only sees part of the week can mislead you.
Sample size depends on three things:
- Your baseline conversion rate. Lower baselines need more visitors to detect the same relative change.
- The smallest lift you care about. This is the minimum detectable effect (MDE). Detecting a 5% lift takes far more traffic than detecting a 20% lift.
- Your confidence and power settings. A common default is 95% confidence and 80% power.
A worked example
Say your store converts at 2.5% and you want to detect a 10% relative lift (from 2.5% to 2.75%) at 95% confidence and 80% power. The standard two-proportion formula gives about 64,200 visitors per version, or roughly 128,400 in total.
With 30,000 visitors a month, that test would take more than four months. If you only care about a 20% lift, the need drops to about 16,800 visitors per version, which is about five weeks of traffic. If you want to detect a 5% lift, it climbs to about 250,800 per version.
The lesson: halving the effect you want to detect roughly quadruples the visitors you need. That is why smaller stores should test bold changes, like a new offer or a rewritten headline, not button colors. Plug your own numbers into the sample size calculator.
Don't stop early
Stopping a test the moment it looks good is the most common way to fool yourself. Ruben tested an authority badge (a doctor or expert endorsement) that showed +17% after week one. After four weeks it settled at +1.66% conversion rate, which was flat. The early lift was noise.
Early results swing because small samples are noisy. A handful of large orders in one group can make it look like a big winner for a few days. Set the sample size and end date before launch, and stick to them.
Low-traffic stores
If your store gets too few orders to reach significance within a few weeks, A/B testing may not be your best tool yet. You can test on a metric higher in the funnel, such as add-to-cart rate, make bigger changes, or lean on customer research and proven patterns from the database until traffic grows.
How to read A/B test results
When a test ends, you are answering two questions: is the difference real, and is it big enough to matter?
Statistical significance and confidence
Statistical significance tells you how unlikely your result would be if the change had no effect. Most tools report it as a confidence level or a "chance to beat control." Frequentist and Bayesian tools phrase it differently, but both are trying to separate a real effect from random noise.
To check a finished test yourself, paste the visitors and conversions for each version into the significance calculator. Our guide to statistical significance explains what the number does and doesn't mean.
Confidence levels vary in practice, so report them honestly. Kasper Dolk logged a +€127,112 revenue lift for an optimized site-wide navigation at 90% confidence, and a +€54,910 lift from a product finder quiz on the collection page at 89% confidence while that test was still in progress. A 90% result is useful evidence, but it carries more risk of being a false positive than a 95% or 99% result.
Results don't always transfer
A win on one store is a hypothesis for yours, not a guarantee. Countdown timers show this well:
- Dave Diederen saw +3.16% RPV (an estimated +$139,029/month) after adding a countdown timer to checkout.
- A second checkout countdown timer test, also from Dave, lost 7.1% RPV.
- On a non-US (European) market, a site-wide countdown timer lost 35.61% RPV.
- Ruben tested a countdown timer across several brands: about +10% conversion rate on a low-priced brand, flat on a high-priced brand, and -9% on another.
Audience, price point, and market change the outcome. That is the strongest argument for testing on your own store instead of copying a competitor.
Check before you ship
- Did the test reach its planned sample size and run full weeks?
- Is the primary metric the one you chose in advance, not one you picked after seeing the data?
- Does the result hold on both mobile and desktop, or is one segment driving it?
- Did conversion rate rise while average order value fell? Check revenue per visitor too.
- Were both groups about the same size? A large mismatch can mean a tracking or redirect bug.
A flat or losing test is not a failure. It stopped you from shipping something that would have hurt revenue. Our list of common A/B testing mistakes covers the traps that make results unreliable.
A/B testing tools for ecommerce
Most stores pick from three types of tools: dedicated testing platforms, Shopify-native options, and page builders with built-in testing. Google Optimize, once the default free choice, shut down on September 30, 2023, so older guides that recommend it are out of date.
| Tool | Type | Pricing (as of November 2026) | Good for | Not for |
|---|---|---|---|---|
| Convert | Dedicated testing platform | From $299/month billed annually (Growth, 100K tested users/month); 15-day free trial | Stores and agencies that want a full testing tool on Shopify or any site | Low-traffic stores or tight budgets |
| VWO (by Wingify) | Testing plus behavior analytics | Tiers on the vendor's pricing page, list prices not shown; 30-day free trial | Teams that want testing, heatmaps, and session recordings in one tool | Teams that only need a simple A/B tool |
| Shopify Rollouts | Built into Shopify | Included; experiments need the Grow plan or higher | Whole-theme tests without third-party scripts | Testing single elements or detailed segments |
Convert. According to Convert's pricing page, as of November 2026 the Growth plan costs $299/month billed annually or $399/month billed monthly, both including 100K tested users per month. The Pro plan is $420/month billed annually. The 15-day trial needs no credit card. The downside for small stores is cost: a store with a few thousand monthly visitors won't run enough conclusive tests to justify it.
VWO. VWO's testing product now sits under the Wingify brand. Its product page lists a no-code visual editor; A/B, multivariate, and split URL tests; and heatmaps and session recordings, with a 30-day free trial and no credit card. Wingify's pricing page shows Growth, Pro, and Enterprise tiers but did not show list prices when we checked, so expect to talk to sales for a quote. It is more than a store needs if it just wants one quick test.
Shopify Rollouts. Shopify's help center says an Experiment rollout tests a treatment against a control, uses a 50/50 split and a 90-day end date by default, and requires the Grow plan or higher. Shopify's changelog describes testing whole themes and checkout and customer account configurations. That makes it a no-extra-cost starting point, but it compares full versions rather than single elements.
Tools mentioned here
For a wider comparison of features and prices, see our roundup of the best A/B testing tools.
If you would rather have an experienced team plan and run the tests, the practitioners behind our database do this work for stores.
Want results like this on your store?
Kasper Dolk at Maison Commerce. Kasper runs Maison Commerce, which builds CRO programs, landing pages and themes for fast-growing ecommerce brands such as ICON and HiStrips.
Questions
What is A/B testing?
A/B testing is a way to compare two versions of a page by showing each to a random half of your visitors at the same time. The version that produces more orders or revenue per visitor wins, so changes are based on evidence instead of opinion.
What is split testing, and is it different from A/B testing?
Split testing usually means the same thing as A/B testing: traffic is split between a control and a variant. Some tools use split URL testing for the narrower case where each version lives at its own URL and visitors are redirected between them.
What is the difference between A/B testing and multivariate testing?
An A/B test compares one changed version against the original. A multivariate test changes several elements at once and tests every combination, which shows how elements interact but needs much more traffic.
How long should an A/B test run?
Run it until it reaches the sample size you calculated before launch, and for at least one or two full weeks so every day of the week is covered. Don't stop early because one version is ahead, since early results are often noise.
How much traffic do you need for A/B testing?
It depends on your conversion rate and the smallest lift you want to detect. At a 2.5% conversion rate, detecting a 10% relative lift at 95% confidence and 80% power takes about 64,200 visitors per version, while a 20% lift needs about 16,800.