You can A/B test a Shopify store in two ways. Shopify Rollouts is built into the admin and runs theme and checkout experiments on the Grow plan or higher. A testing app such as Intelligems, Shoplift or Convert handles the tests Rollouts can't judge well, like prices, shipping rates and revenue per visitor. Most stores start with Rollouts for theme changes and add an app once they want to test money: prices, shipping and offers.
This guide covers what Rollouts can and can't do, which type of test fits which question, how to set up a first test, and the mistakes that show up again and again in our database of real Shopify A/B tests.
Native Shopify Rollouts: what it can and can't test
Rollouts arrived with Shopify's Winter 2026 Edition. Shopify's help center describes a rollout as a scheduled set of changes to your main theme, checkout and accounts pages, discounts, product catalogs, or a mix of these.
There are three rollout types:
- Launch: merges your changes into the live store for good.
- Event: publishes changes for a set period, then rolls them back. This is useful for a holiday theme.
- Experiment: splits traffic between a control and a treatment so you can measure the difference. This is the A/B test.
Plan and theme requirements
According to Shopify's requirements page, rollouts are available on the Basic plan or higher. Experiments need the Grow plan or higher. As of November 2026, Shopify's pricing page lists Grow at $105/month billed monthly, or $79/month billed annually.
The same page lists some hard limits. Vintage themes can't be used. Liquid templates can't be modified as part of a rollout. Theme changes only affect the online store, not headless or custom storefronts.
What it measures
This is the main weakness. Shopify's rollout analytics page lists four metrics for a theme experiment: conversion rate over time, bounce rate, reached-checkout rate and add-to-cart rate. A checkout experiment gets one metric: checkout conversion rate.
That page says nothing about statistical significance, confidence or probability to win. It also has no revenue per visitor or profit metric. A change can raise conversion rate and still lose money if it pulls down order value, so for anything touching price or offers you need revenue per visitor.
Convert's write-up of the launch makes a similar point. It describes Rollouts as built mainly for safe deployments, not for conversion optimization, and notes that testing even a headline means duplicating the theme.
Good fit: testing a full theme redesign, a new homepage layout or a checkout configuration, on a store that already pays for Grow. Poor fit: price tests, shipping-rate tests, small copy tests, and anyone who needs revenue per visitor or a clear win probability.
Theme tests vs template tests vs price tests
Pick the test type based on what your idea changes. Each type answers a different question and needs different tools.
Theme tests
A theme test sends half your visitors to a second copy of your theme. It is the right choice for redesigns and big layout changes, and it's the kind of test Rollouts handles best.
Kasper Dolk ran a new theme with a BOGO offer against the old theme and saw +44.54% conversion rate, with a 99.7% probability to beat the old theme. That is a big win, but note that the variant changed the theme and the offer together. You know the package won, not which part did it.
Template and page tests
A template test changes one page type, such as the product template or one landing page, and leaves the rest of the theme alone. Apps like Shoplift and Intelligems can test templates without you duplicating the whole theme.
Oliver Kenyon rebuilt a landing page natively in Shopify and tested it against the same page built in a page builder. The native version won by +9.9% conversion rate. The design was the same, so the build method was what moved the number.
Smaller product page changes fit here too. Dave Diederen reports that showing the price directly below the product title had a ~90% win rate across 20+ brands.
Price, shipping and offer tests
These tests change what shoppers pay. They have to be judged on revenue per visitor or profit per visitor, so a conversion-rate-only report isn't enough. They're also where the biggest swings in our database are.
- Dave Diederen tested shipping price levels, and a $9.49 shipping fee beat the control by +43.81% RPV.
- In another store, Dave Diederen removed the free-shipping threshold and charged shipping on every order for +9.17% RPV.
- The Elevate A/B Testing team reports that cutting a price by 30% produced a 2x conversion rate.
- Offers can go badly wrong. Milos changed a 2-pack offer to "Buy 2 Get 1 Free" at the same price and saw -52.38% RPV, an estimated -$104,679/month.
Charging for shipping looks like it should lose, and lowering prices looks like it should win. You only know by testing it with a tool that tracks revenue. Our price testing guide covers the method in more detail.
@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen
@elevate_ab
@CROguy_Shopify A/B testing apps compared
Here is how Rollouts compares with three common apps. Prices are entry-level plans as of November 2026, taken from each vendor's pricing page. We haven't used these tools ourselves, so this summary comes from vendor docs and public reviews.
| Tool | Starting price | Theme / template tests | Price and shipping tests | Not for |
|---|---|---|---|---|
| Shopify Rollouts | Included with Grow plan ($105/month monthly) | Theme and checkout experiments | No revenue metrics listed in its analytics | Price tests, small copy tests, anyone who needs win probabilities |
| Intelligems | From $69/month (Build Your Own plan) | Yes (Smart Content) | Yes, as paid add-on modules; price testing isn't in the free trial | Stores that only want simple theme tests |
| Shoplift | From $99/month (Core, up to 100k visitors) | Yes, theme and template tests | Price testing on the $299/month Advanced plan | Shipping-rate testing on the entry plan |
| Convert | From $399/month monthly, or $299/month annual (Growth) | Yes, including a newer Shopify theme testing feature | Yes, Shopify price and shipping testing | Small stores with a tight tool budget |
Intelligems lists a 14-day free trial, as does Shoplift. Convert lists a 15-day free trial. For the full breakdown, see our Shopify A/B testing apps roundup and the Intelligems vs Shoplift comparison.
Setting up your first Shopify A/B test
The steps below are the same whatever tool you use.
- Write one hypothesis. Name the change, the page and the metric. For example: "Moving the price below the title on product pages will raise revenue per visitor."
- Check you have enough traffic. Put your conversion rate and the lift you hope to detect into our sample size calculator. If the answer is months of traffic, test a bigger, bolder change.
- Build the variant. In Rollouts, create an experiment from your published theme and edit the treatment copy. In an app, build the variant in the app's editor or as a duplicate template.
- Preview it on mobile and desktop. Check that cart drawers, apps and tracking still work in the variant.
- Pick the primary metric before launch. Use conversion rate for layout tests. Use revenue per visitor or profit per visitor for anything that touches price, shipping or offers.
- Run full weeks. Run at least two full business cycles, usually two weeks or more, and don't stop early. Our guide on how long to run an A/B test explains why.
- Roll out or roll back. In Rollouts, a winning treatment can be launched to everyone. Record the result either way, because losing tests teach you what your shoppers don't want.
Common Shopify testing mistakes
Copying a winner from another store
The same idea can win in one store and lose in the next. Dave Diederen's checkout countdown timer gained +3.16% RPV in one store. A checkout countdown timer in another store lost -7.1% RPV, and one for a European market lost -35.61% RPV. Kasper Dolk went the other way: removing a countdown timer from checkout gave CVR +25.61% and RPV +26.18%.
Calling a test after the first week
Ruben's authority badge test looked like +17% after week one. By week four it had settled at +1.66% CVR, which is flat. Early results swing a lot, so decide the run length before you start.
Testing several changes at once
Theme tests often change ten things at the same time. If the variant wins you can ship it, but you won't know which change worked. When you need to know what caused a result, test one change at a time.
Judging money changes on conversion rate
Rollouts only reports conversion-style metrics. Offer and shipping tests need revenue per visitor or profit per visitor, or a "winner" can cost you margin. Pick the metric before the test starts.
Removing things that look like friction
Dave Diederen removed an email popup and lost -20.51% RPV. Dave Diederen also skipped the cart and sent shoppers straight to checkout, which lost -10.23% CVR. Test removals the same way you test additions.
For more ways tests go wrong, see A/B testing mistakes. For ideas worth testing first, see our Shopify CRO guide.
@DaveDiederen
@kasper_dolk@DaveDiederen@DaveDiederen@DaveDiederen
@Rubendgrf