Amazon A/B testing is done with Manage Your Experiments, a free Seller Central tool that shows two versions of a listing's title, images, bullet points, description or A+ Content to randomly split groups of shoppers and reports which version sold more.
You need Brand Registry and an ASIN with enough recent traffic, and Amazon recommends running each experiment for 8 to 10 weeks.
The tool is easy to start. The hard part is picking changes big enough to detect and reading the results correctly, because Amazon's default bar for calling a winner is much lower than most testing teams use.
How Amazon split testing works
Amazon split testing is not a before-and-after comparison.
ChannelX's report on the launch explained that shoppers who view the ASIN are randomly split into two groups, and each group sees only Version A or Version B for the whole experiment.
Both versions run at the same time, so seasonality and ad spend hit both groups equally.
The experimental content follows the shopper. An experimental title, for example, shows in search results, on the detail page and in the cart.
That matters: a title test measures the effect on clicks from search as well as on conversion from the page.
It also means a few things you would do on your own site are off the table:
- You can't choose the traffic split, the audience or the device.
- You test two versions, not five. Multi-attribute experiments change several elements at once, which tells you whether the bundle won, not which piece did the work.
- You can't edit content once an experiment is running. Amazon's help page says you have to cancel and start a new experiment to change it.
- All content must pass Amazon's normal listing rules. A main image with a non-white background fails validation in categories that require white.
If you already run tests on your own store, the logic is the same as in our A/B testing guide.
What changes is how much control you have over setup and analysis.
What to test first, based on product page tests
Amazon doesn't publish individual experiment results, and almost all tests in our database ran on brands' own stores rather than on Amazon.
So read the tests below as evidence about which product page elements move sales, not as Amazon results.
They point to the same three places Manage Your Experiments lets you test.
The first image. It is the one shoppers see in search grids, which on Amazon is most of the decision.
Dave Diederen tested a new first product image and reported +10.26% conversion rate and +10.44% revenue per visitor.
Elevate A/B Testing moved The Cornish Company's second photo into the lead slot, and revenue per visitor went from £3.32 to £3.60 (+8.4%), with no sample size or significance reported.
Convertibles tested four versions of a first gallery image and found that a bold claim plus the product in use added $15,824 a month.
On Amazon, main images in many categories must be the product on white, so claims, lifestyle shots and overlays belong in the secondary images.
Dave Diederen's test of images that include USPs, testimonials, guarantees and delivery info reported +7.17% conversion rate, and Oliver Kenyon's USP images in the gallery added $10K a month.
Image changes can also lose. Clean Commit replaced model photos with lifestyle images on three products and saw -7.2% conversion rate over 8,808 visitors, with an 84.9% probability that the control was better.
That is not a significant result, but it is a useful warning against assuming lifestyle images always win.
More examples are on our product images page.
The title. Dave Diederen changed a product title to a benefit-led "Achieve [outcome] with [product]" format and reported +15.61% conversion rate and +16.92% revenue per visitor.
On Amazon, a title also carries search keywords, so a benefit-led rewrite has to keep the terms shoppers search for.
Amazon's experiment help page lists a 75-character limit for product titles, including spaces.
The bullets. ConvertPolo added short benefit bullets to a socks product page that had only basic information, with +18.16% conversion rate and +16.37% revenue per visitor.
DRIP Agency rewrote KoRo's descriptions and bullets to make benefits and taste quicker to grasp, and revenue per user went from €27.41 to €28.33 (+3.4%).
@DaveDiederen@DaveDiederen
Elevate_ab
DRIP Agency
Clean CommitAmazon Manage Your Experiments: setup and settings
According to Amazon's Manage Your Experiments page, you need a Professional selling account, a brand enrolled in Brand Registry, Brand Representative access, and a product with enough traffic in recent weeks.
A+ Content and Brand Story experiments also need that content already published.
Low-traffic ASINs may not appear in the list of products you can test.
Setup takes a few minutes in Seller Central:
- Open the Manage Your Experiments dashboard and pick the experiment type.
- Choose the reference ASIN. For variation families, you must also submit content for the parent ASIN.
- Name the experiment and write a hypothesis. Amazon asks for one, and it's the part that lets you reuse the result on other products.
- Pick the duration and whether to auto-publish the winner.
- Create Version B, usually by duplicating Version A and editing it, then submit for content validation. The start date may be several days out.
The settings that matter most, from Amazon's help documentation:
| Setting | What Amazon says | What it means for you |
|---|---|---|
| To significance | Ends when results are "statistically relevant," in as little as four weeks. Title, image and bullet experiments only. | Faster, but you don't control the stopping rule. |
| Set duration | You choose start and end dates; eight weeks is recommended. | The only option for A+ Content and Brand Story tests. |
| Auto-publish | Publishes the winner only if it has at least a 66% probability of beating the other version. Title, image and bullet experiments only. | A low bar. Consider turning it off and deciding yourself. |
| Simplified title test | Editing a title can create an experiment automatically, with the new title as Version B and the winner auto-published. | Opt out before seasonal title edits, as Amazon itself suggests. |
How to read the results
Results update weekly. Amazon reports units sold, sales, conversion rate, units sold per unique visitor and sample size, plus the probability that each version is better and a projected one-year sales impact.
The probability is a Bayesian-style statement, not a p-value.
If Amazon says Version B has a 74% probability of being better, it is saying there is a 26% chance Version A is actually the better one.
Our page on Bayesian vs frequentist testing explains the difference.
That's why the 66% auto-publish threshold deserves caution.
A version published at exactly 66% still has about a one-in-three chance of being no better than what it replaced.
On a single listing that may be an acceptable bet.
If you run dozens of experiments and publish everything above 66%, many of your changes will be noise.
Many teams wait for 90% to 95% before acting on high-stakes content such as the main image.
Treat the projected one-year impact as an optimistic estimate.
Winning tests tend to overstate the true effect, a pattern we cover in the winner's curse.
Amazon's own marketing pages say the tool can lift sales "up to 20%" on the tool page and "up to 25%" on its seller blog. Those are upper bounds, not typical results.
Two practical rules help. Make the versions very different: Amazon's help page says the bigger the difference, the more likely a result is real and not random chance.
And let experiments run at least a few full weeks, so weekday and payday patterns even out.
Our guide on how long to run an A/B test explains why stopping the moment a number looks good inflates false wins.
A+ Content A/B tests
An A+ Content A/B test compares two versions of the enhanced content block below the bullets, or of the Brand Story carousel.
Version A must already be approved. You build Version B in A+ Content Manager, and both versions need approval before the experiment can start.
A+ tests have tighter limits than title, image and bullet tests.
They can't use "to significance" or auto-publish, so you set a fixed duration (Amazon recommends eight weeks) and publish the winner yourself.
Fewer shoppers scroll far enough to see A+ Content than see the title or main image, so effects are diluted and harder to detect.
Amazon's advice is to test different modules or a different module order, not small copy edits.
A few ideas backed by tests on brand sites:
- A comparison chart. Dave Diederen added a value comparison chart to a product page and reported +9.4% conversion rate and +7.68% revenue per visitor. A+ Content has comparison modules for this.
- Benefit-first modules. The bullet and title tests above suggest leading with outcomes over specs. In A+ terms, that means moving a benefits module above the technical details.
- Video, with caution. Results are mixed. One Click reported +10% add-to-carts from product videos on Readers.com, while Dave Diederen saw -1.60% revenue per visitor after replacing written reviews with only product videos.
Keep a simple log of every Amazon experiment: hypothesis, versions, dates, probability and what you published.
Amazon's dashboard keeps the history, but a written log lets you carry what you learn to other ASINs and to your own store, where you can test the same idea with more control.