To run an A/B test, pick one page and one primary metric, write a hypothesis, work out how many visitors you need, build one variant, check it on every device, then let it run to that sample size without stopping early. When it ends, judge it on the metric you picked up front and write down what you learned, win or lose.
That is the whole A/B testing process. The rest of this guide walks through each step, with real results from our database of 285 ecommerce tests showing what goes right and wrong at each stage.
1. Pick a page and a metric
Start where traffic and money meet. For most stores that is the product page, the cart or checkout, because small changes there touch every buyer. A page with a few hundred visitors a month will not give you a readable result in any sane amount of time.
Then choose one primary metric before you build anything. Conversion rate (CVR) tells you whether more people bought. Revenue per visitor (RPV) tells you whether you made more money, which catches changes that lift orders but shrink basket size.
The metric choice can flip the verdict. When Dave Diederen pre-checked the subscription option on a product page, the result was -9.17% RPV upfront alongside a +1,295% jump in subscription share. Judged on day-one revenue it lost; judged on subscribers it won big. Decide which number matters before launch, not after.
Pick a couple of guardrail metrics too, such as average order value or refund rate, so a "win" on the main metric does not hide damage somewhere else.
2. Write the hypothesis
A hypothesis says what you will change, for whom, and why you expect it to move your metric. A simple format: "Because we saw [evidence], we believe [change] for [audience] will increase [metric]." Our hypothesis guide has more templates.
The "because" part matters most. Good evidence comes from heatmaps, session recordings, surveys and support tickets. Dave Diederen rewrote the above-the-fold USPs around the points heatmaps showed buyers engaging with and saw +6.23% CVR. The idea came from data, not a hunch.
Also check whether the idea has worked elsewhere, and whether it has failed. Countdown timers are a good warning. Ruben tested a countdown timer across several brands and got about +10% CVR on a low-priced brand, a flat result on a high-priced brand, and -9% CVR on another. The same idea can win, lose or do nothing depending on the store, which is why you test instead of copying.
If you have more ideas than traffic, rank them first by expected impact, confidence in the evidence and effort to build.
3. Size the sample
Before launch, calculate how many visitors each variant needs. You need four inputs: your current conversion rate, the smallest lift worth detecting (the minimum detectable effect, or MDE), a significance level (usually 95%) and statistical power (usually 80%).
Here is a worked example. A product page converts at 3%, and you want to detect a 10% relative lift (3.0% to 3.3%). At 95% significance and 80% power, the standard two-proportion formula gives roughly 53,200 visitors per variant, or about 106,000 in total. If you only care about a 15% lift (3.0% to 3.45%), that drops to roughly 24,200 per variant.
Plug your own numbers into our A/B test sample size calculator, then divide by your daily traffic to get the run time. Round up to whole weeks so every weekday and weekend is covered equally. Our guide on how long to run an A/B test goes deeper.
If the answer is several months, test bigger changes so the expected lift is larger, or test on a higher-traffic page first.
4. Build the variant
Change one idea per test. You can touch several elements if they all serve the same hypothesis, but if you change the headline, the price and the images at once, you will not know which one did the work. If you want to test combinations, that is a multivariate test, and it needs far more traffic.
Make the change big enough to be noticed. Small cosmetic tweaks often produce no measurable change at normal store traffic. Dave Diederen placed a review rating above the product title and the result was flat, with no CVR change.
Most stores build variants in a testing tool's visual editor, with custom code, or with a page builder. Our roundup of A/B testing tools compares the options. Two common picks for Shopify and other platforms are below.
Tools mentioned here
You only need one A/B testing tool to copy this result. Here's where we'd start.
Also works well
Some of these are affiliate links. We only list tools practitioners actually use to run tests like these.
Neither is the right fit for everyone. Both are built around a JavaScript snippet and a visual editor, so a store that only wants to test prices or shipping fees inside Shopify may prefer a Shopify-native app. And a store with very low traffic will pay for a tool it cannot get readable results from yet.
5. QA and launch
A broken variant does not test your idea; it tests a bug. Before real visitors see it, check every variant on desktop and mobile, in the main browsers, logged in and logged out, and with a product in the cart.
Convert's QA guide is a useful model even if you use another tool. It recommends previewing each variation, then using force-variation links in incognito windows with a QA-only audience, then confirming in the report that visitor counts go up and goals fire. It says these checks should be run on each device type you plan to target.
A short pre-launch checklist:
- The variant renders correctly and does not flicker the original first.
- Add to cart, checkout and any discounts still work in both versions.
- Your primary and guardrail goals fire once per conversion, not twice.
- The traffic split and targeting (device, country, new or returning) match your plan.
- Internal staff and QA traffic are excluded from the results.
After launch, check the split in the first day or two. If you set 50/50 but one side has noticeably more visitors, you may have a sample ratio mismatch. VWO's SRM explainer lists causes such as JavaScript errors, blocked cookies and device bias, and recommends a chi-square test, where a p-value below 0.05 signals SRM. Fix the cause and restart; do not try to analyze around it.
6. Set stop rules
Write your stop rules before launch: the sample size from step 3, a minimum of one to two full weeks, and the significance level you will accept. Then do not stop because the dashboard looks exciting on day three.
Checking early and stopping on the first good number inflates false positives. Evan Miller's "How Not To Run an A/B Test" shows that if you peek at a test ten times, what you think is 1% significance is actually 5%. The fix it recommends: decide on a sample size in advance and wait until the experiment is over.
Our database has a clear example. Ruben added a doctor and expert endorsement badge to a product page. After week one it looked like +17%. After four weeks it settled at +1.66% CVR, essentially flat. The week-one number was noise.
Want results like this on your store?
Ruben at Tailored Commerce. Ruben runs CRO at Tailored Commerce for brands like Hears, Moov and FLÂNEUR.
It is fine to stop early for one reason: the variant is broken or clearly hurting revenue badly. Set that threshold in advance too.
7. Analyze and document
When the test reaches its sample size, read the primary metric first, then the guardrails, then segments such as mobile versus desktop. Use the significance level you set before launch, not a new one.
Report the confidence along with the lift, as the practitioners in our database do. Kasper Dolk's testimonial test reported +7.22% profit per visitor with a 94.4% probability to beat control. Kasper Dolk's navigation redesign reported a +€127,112 revenue lift at 90% confidence. Those confidence levels tell the reader how much weight the result can carry.
Treat segments as ideas for the next test, not as new winners. If you slice a flat test ten ways, one slice will look like a win by chance.
Losers and flat results are worth writing down too. Dave Diederen added small customer profile icons next to the star rating and saw -3.84% RPV. The same practitioner ran a checkout countdown timer that won +3.16% RPV on one store and another that lost 7.1% RPV on a different one. Without a record, a team will rerun the losing idea a year later.
Want results like this on your store?
Dave Diederen at Syntra Labs. Dave builds and tests Shopify stores at Syntra Labs, and shares most results in public.
For each test, log the page, hypothesis, evidence, screenshots of both versions, dates, sample size, primary and guardrail results, confidence, and the decision. Then use what you learned to pick the next test. Our list of common A/B testing mistakes is a good final check before you start the next one.
Questions
How do you do A/B testing step by step?
Pick a page and one primary metric, write a hypothesis based on evidence, calculate the sample size, build one variant, QA it on every device, run it to the planned sample size without stopping early, then analyze the result and document it.
How long should an A/B test run?
Until it reaches the sample size you calculated before launch, and for at least one to two full weeks so every day of the week is covered. Divide the required visitors by your daily traffic to estimate the run time.
How many visitors do I need to run an A/B test?
It depends on your conversion rate and the lift you want to detect. At a 3% conversion rate, detecting a 10% relative lift at 95% significance and 80% power takes roughly 53,200 visitors per variant.
How do I set up an A/B test on my store?
Install a testing tool, create the variant in its editor or with code, set the traffic split and targeting, add your primary and guardrail goals, then preview and QA every variant on desktop and mobile before you launch.
Can I stop an A/B test early if it is winning?
No. Early leads are often noise, and repeatedly checking and stopping on a good number inflates false positives. Only stop early if the variant is broken or causing a large loss you defined in advance.