Plan most store A/B tests around a relative minimum detectable effect (MDE) of 10% to 20% on your main metric, and only go lower if you have hundreds of thousands of visitors per variant. The MDE is the smallest lift your test can reliably catch, so it should be both a lift you could realistically get and one your traffic can detect in two to six weeks.
Below we explain what MDE means, show the lifts real stores have published in our database of 285 ecommerce tests, and walk through the math so you can pick a number before you launch.
What minimum detectable effect means
The minimum detectable effect is the smallest true difference between control and variant that your test is powered to detect. Optimizely defines it as "the relative minimum improvement over the baseline you are willing to detect in an experiment to a certain degree of statistical significance."
"Powered to detect" has a specific meaning. With the usual 80% power, if the variant really is better by the MDE or more, your test will flag it as significant about 80% of the time. If the real effect is smaller than the MDE, the test will usually miss it and call it flat.
Relative vs absolute MDE
MDE can be written two ways, and mixing them up is the most common planning error. Evan Miller's sample size calculator lets you switch between both.
- Relative MDE: a percentage of the baseline. A 10% relative MDE on a 2.5% conversion rate means detecting a move to 2.75%.
- Absolute MDE: percentage points. A 0.25-point absolute MDE on the same 2.5% baseline is the same test, written differently.
Most testing tools, and most published results, use relative lifts. When Dave Diederen reports +10.74% CVR, that is a relative change. We use relative MDE for the rest of this page.
MDE is a planning input, not a result
You pick the MDE before the test starts, together with your significance level and power. It sets the sample size and therefore how long the test runs. It is not a prediction of what the variant will do.
A good way to choose it, as Optimizely's guide puts it, is to ask what improvement would justify the cost of building and shipping the change, then check whether your traffic can detect that improvement in a sensible time.
Realistic lifts from the database
An MDE only makes sense if real changes can reach it. So we looked at every result in our database that reports a percentage lift in conversion rate or revenue per visitor. That gave 117 results, 111 of them positive.
| Published lift (CVR or RPV) | Number of winning tests | Share of winners |
|---|---|---|
| Up to 5% | 19 | 17% |
| 5% to 10% | 38 | 34% |
| 10% to 20% | 38 | 34% |
| Over 20% | 16 | 14% |
The median published win was 9.77%. That sounds encouraging, but read it with care: these are tests practitioners chose to share, and shared tests skew toward winners. Most tests in the wild do not win at all, as our A/B test win rate research covers.
Small tweaks usually land below 10%
Single-element changes tend to produce single-digit lifts. Dave Diederen saw +3.16% revenue per visitor after adding a countdown timer to checkout. Dave also got +1.36% RPV from making the star rating near the title static, and Ruben saw +1.66% CVR from adding an expert authority badge.
If you planned any of those tests around a 15% MDE, a real win of 2-3% would most likely come back as "no significant difference." That is not a failed idea; it is an underpowered test.
Bigger changes can clear 10-20%
Changes to the buy box, offer framing or page structure show up more often in the 10-20% band. Dave Diederen saw +10.74% CVR after making the bundle buy box more compact, and +7.48% CVR after putting the money-back guarantee inside the add-to-cart button.
@DaveDiederen@DaveDiederen@DaveDiederen@DaveDiederen
@CROguy_@DaveDiederenRedesigns are where 20%+ lifts live
The largest results in the database are full rebuilds. Milos saw +128% RPV after rebuilding a best-seller product page. Dave Diederen saw +72.15% RPV from redesigning the homepage hero, and Oliver Kenyon saw +47% CVR after sending cold traffic to a listicle-style presell page. See more in our list of the biggest A/B test wins.
The trade-off: a redesign changes many things at once, so you learn less about which part worked. Low-traffic stores often accept that because a big swing is the only kind they can detect.
The same lift can go negative
Your MDE also applies to losses. The same checkout countdown idea that won once lost 7.1% RPV in another of Dave Diederen's tests, and lost 35.61% RPV for a European store. A test powered to see a 10% change will catch drops like these too, which is half the point of testing.
The MDE vs traffic trade-off
The required sample size grows with the inverse square of the MDE. Halve the effect you want to detect and you need roughly four times the visitors. Optimizely's guide gives an example: on a 10% baseline, cutting the MDE from 10% to 3% raises the sample from 12,243 to 185,661 visitors per variation.
Here is what that looks like for a typical store. We calculated these numbers with the standard two-proportion formula at 95% significance (two-sided) and 80% power, with one control and one variant.
| Relative MDE | Visitors per variant at 2% CVR | At 2.5% CVR | At 3% CVR |
|---|---|---|---|
| 5% | 315,206 | 250,846 | 207,938 |
| 10% | 80,682 | 64,199 | 53,211 |
| 15% | 36,693 | 29,193 | 24,193 |
| 20% | 21,109 | 16,792 | 13,914 |
| 30% | 9,798 | 7,792 | 6,455 |
Two things stand out. A 5% MDE is out of reach for most stores, since it needs half a million visitors in total. And a higher baseline conversion rate makes every test cheaper, which is one reason to test on high-intent pages like the cart and product page.
How to pick your number
- Start from weekly traffic. Count the visitors who actually reach the tested page, not sitewide sessions.
- Cap the run time. Two to six full weeks is a common range; our guide on how long to run an A/B test explains why.
- Find the smallest MDE that fits. Divide the traffic you will get in that window by two (for two variants) and read off the MDE.
- Match the idea to the MDE. If the smallest MDE you can afford is 20%, a button-color test is a poor use of your traffic. Test something bolder.
If even a 30% MDE does not fit, read our guide to testing with low traffic before running anything.
Metric choice changes the math
Revenue per visitor varies more from visitor to visitor than conversion rate does, so an RPV test usually needs more traffic than a CVR test at the same MDE. If you plan on RPV, treat the table above as a floor. Our A/B testing metrics guide covers when each one fits.
Calculator and worked example
Our A/B test sample size calculator does this math for you: enter your baseline conversion rate, MDE, significance and power, and it returns visitors per variant.
The formula
For two variants with baseline rate p1 and target rate p2 = p1 × (1 + MDE), the visitors needed per variant are:
n = (z(α/2) × √(2 × p̄ × (1 − p̄)) + z(β) × √(p1(1 − p1) + p2(1 − p2)))² / (p2 − p1)²
Here p̄ is the average of p1 and p2, z(α/2) is 1.96 for 95% significance, and z(β) is 0.84 for 80% power. A quick shortcut that lands close is n ≈ 16 × p1 × (1 − p1) / (p2 − p1)².
Worked example
Say your product page gets 20,000 visitors a week and converts at 2.5%.
- 10% MDE: the target rate is 2.75%, a 0.25-point change. The formula gives 64,199 visitors per variant, or 128,398 in total. At 20,000 a week, that is about 6.4 weeks, so plan seven full weeks.
- 20% MDE: the target is 3.0%. You need 16,792 per variant, 33,584 in total, which is under two weeks. Run two full weeks to cover weekday and weekend behavior.
- 5% MDE: 250,846 per variant, about 25 weeks. Not practical for this store.
The shortcut checks out: 16 × 0.025 × 0.975 / 0.0025² is about 62,400, close to the exact 64,199.
Given the database numbers above, this store would do well to plan around a 15-20% MDE and save its traffic for bigger changes such as buy box, offer or page structure tests.
Running the test
Most testing tools ask for or report an MDE in their own sample size planning. Convert's Growth plan costs $299 a month billed annually, or $399 month to month, as of November 2026, and includes 100,000 tested users a month with a 15-day free trial. That fits stores with enough traffic to test at a 10-20% MDE.
It is not for every store. If your tested page gets only a few thousand visitors a month, you cannot reach a useful MDE in any tool, and the fee buys you little until traffic grows. Shopify-only stores may also prefer a native app; see our Shopify A/B testing apps roundup or the full Convert review.