A/B Testing vs Multivariate Testing: When to Use Each

A store should run a multivariate test (MVT) instead of an A/B test only when it wants to know how two or more page elements work together, and it has enough traffic to fill every combination. For most Shopify stores that means A/B testing one change at a time, and saving MVT for high-traffic pages like the homepage hero or a best-selling product page.

Below we define both methods, show the traffic math, list the cases where MVT is worth it, and explain where split URL tests fit. Throughout, we use real results from the LiftArchive test database.

Definitions: A/B testing vs multivariate testing

An A/B test shows half your visitors the current page (the control) and half a changed version. You compare one metric, such as conversion rate or revenue per visitor, and learn whether the change helped. An A/B/n test is the same idea with more than one challenger.

A multivariate test changes several elements at once and tests every mix of them. VWO's help center defines it as a test "wherein multiple elements on a single page are changed to determine the optimal combination of those changes" (VWO help center).

The number of versions is the number of options for each element multiplied together. Two headlines and three hero images give 2 x 3 = 6 combinations. Convert calls this a full factorial design, because "every combination gets measured" (Convert).

A quick example

Say you want to test a product title and a review rating on the product page. On our database, Dave Diederen saw +15.61% CVR from a benefit-led "Achieve [outcome] with [product]" title. In a separate test, Dave Diederen placed a review rating above the product title and saw no CVR change, while in another Dave Diederen test, a star rating and review count added above the title brought +8.14% CVR.

Results like these raise a fair question: does the rating help more with one title than another? An A/B test answers "is the new title better?" A multivariate test with 2 titles x 2 rating placements (4 combinations) answers "which title and rating placement work best together?"

Traffic needs: why MVT is hard for most stores

Every combination in an MVT is its own variant, and each one needs enough visitors and orders to reach statistical significance. Traffic gets split evenly across all of them by default, so each new element multiplies the visitors you need.

Convert gives a rule of thumb of at least 1,000 users and 100 conversions per combination, and says a 6-combination test needs "roughly 6,000 minimum weekly users as a starting point" (Convert). VWO's guide says to use MVT only "when your website receives high traffic" and suggests breaking a slow MVT into separate A/B tests (VWO help center).

Work out the math before you build

Here is how fast the minimum grows using Convert's 100-conversions-per-combination floor:

• 2 elements x 2 options = 4 combinations, so at least 400 conversions
• 3 elements x 2 options = 8 combinations, so at least 800 conversions
• 2 elements x 3 options = 9 combinations, so at least 900 conversions
• 3 elements x 3 options = 27 combinations, so at least 2,700 conversions

Those are floors, not targets. Smaller effects need far more. Run your own numbers in our A/B test sample size calculator: work out the sample per variant, then multiply by the number of combinations. If the result means a test of more than four to six weeks, see how long to run an A/B test and cut the design down.

If the page gets a few hundred orders a month, skip MVT. Our guide to A/B testing with low traffic covers what to do instead, like testing bigger changes that can show a larger minimum detectable effect.

When MVT makes sense (and when an A/B test is better)

MVT is worth it when three things are true: the page has high traffic, you have two or three elements you think affect each other, and you want to tune the page rather than rethink it.

Good fits for multivariate testing

• A homepage hero with high traffic. Headline, image and CTA copy sit together and are read together. Dave Diederen saw +8.15% RPV from changing homepage CTA copy to "Shop Now" and +52.67% RPV from replacing a hero video with a static image and benefit callout. A store with the traffic could test CTA copy and hero media together to see whether the two interact.
• A best-selling product page's title and social proof. As in the example above, title wording and rating placement are close together and plausibly interact.
• Tuning a page that already won. After a redesign wins, MVT can sort out which of its parts are doing the work.

When an A/B test is the better choice

• You want to test a bold redesign. Many of the biggest wins in our database bundled several changes into one A/B variant. Dave Diederen saw +72.15% RPV from a homepage hero redesign, and Milos (CROguy_) saw +81% CVR from a product page redesign with a lifestyle image, benefit headline, four bullets, testimonials and bundles. Those were not split into dozens of combinations, and they did not need to be.
• You are testing options of one element. That is an A/B/n test, not an MVT. Dave Diederen tested +2px and +4px font size variants for an older audience and saw +3.82% RPV. For Dave Diederen, swapping the model in the first product image (blonde vs brunette) was worth +$18,828/month, also with one element changing.
• You have not found a winner yet. Early on, test the biggest ideas first. A simple prioritization score helps you pick.