The most surprising A/B test results are the ones where the expert answer lost: a long Medicare page that beat a short one by 638%, a seven-step form that beat a one-page form by 154%, and a free extra unit in a bundle that cut revenue per visitor by 52.38%.
The other big surprise is size. One word in a headline nearly doubled purchases, and making a button's text 2px bigger lifted clicks 42%.
How we picked: we went through the 1,824 results in the LiftArchive test database and took real A/B tests whose result ran against common advice, or where a tiny change moved a big number.
Every number below is copied from the test record, with who ran it. We did not run these tests ourselves.
This page avoids the tests on our lists of best practices that failed and losing A/B tests, so read those for countdown timers, trust bars and popups.
Browse the rest of our A/B testing research for more.
How we ranked them
We ranked by how strongly each result contradicts what most people would predict, then by size of effect.
Tests with clear stats and sales or revenue metrics rank above click-only tests.
Where a test is thin (no sample size, wide ranges, not final), we say so in its entry.
Ranked table
| Rank | Test | Change | Result | Who ran it |
|---|---|---|---|---|
| 1 | A long Medicare landing page | Short page to long, advisor-led page | +638% leads | MECLABS Institute |
| 2 | Button label | 'Download' + price to 'Price Guide' | +620.9% clickthroughs | PriceCharting |
| 3 | Bundle offer | 2-pack to 'Buy 2 Get 1 Free', same price | -52.38% revenue per visitor | Milos |
| 4 | Inquiry form | One page to seven steps | +154% conversion rate | HomEquity Bank |
| 5 | Landing page sections | FAQ removed | +62% sales | AssessmentDay |
| 6 | Headline | Added the word "supplement" | +89.97% purchases | Source Patrol |
| 7 | Button text | 'Add To Bag' to 'Add To Cart' | +81.4% purchases | Conversion Fanatics |
| 8 | Product page buttons | Express checkout removed | +66.18% revenue per visitor | Blend Commerce |
| 9 | Subscription checkout | Three steps to one page | -29% subscriptions | MECLABS |
| 10 | Registration form | Shorter first page | Paid 0.97% to 0.72% | MarketingExperiments |
| 11 | Annual price | $69.95 to $50.00 | About 15% less revenue | MarketingExperiments |
| 12 | Sign-up offer | Free-forever to 14-day trial of paid plan | +268% paid sign-ups | Acuity Scheduling |
| 13 | App paywall | In-app purchase to web checkout only | 27.0% to 18.1% initial conversion | RevenueCat |
| 14 | App paywall | Close button removed | Refunds outweighed extra trials | BoldVoice |
| 15 | Paywall prompts | About half as many, better timed | +81% primary offer flow | Wizz |
| 16 | Sign-up page design | Plain vs branded | Plain won by 51% | MarketingExperiments |
| 17 | Homepage CTA | Text 13px to 15px | +42% CTA clicks | FENTY |
| 18 | Homepage hero | Video to static image | +52.67% revenue per visitor | Dave Diederen |
| 19 | Shipping fee | Three fee levels tested | +43.81% revenue per visitor at $9.49 | Dave Diederen |
| 20 | Header and footer | Decluttered header, fuller footer | -16.3% conversion rate | Clean Commit |
The 20 most surprising A/B test results
1. A long Medicare landing page beat a short one by 638%
The change: HealthSpire, an Aetna call-center business, replaced a short, simple Medicare page with a much longer one.
The long page added value copy for each product, framed the phone agents as trusted advisors, used agent photos and ran a Q&A with an agent.
The result: +638% leads (calls and requests to the call center), run by MECLABS Institute.
A/B test; several copy and layout changes went out together.
Why it likely worked: Medicare is a confusing, high-stakes choice.
The short page asked people to pick up the phone before it had given them a reason to trust the person on the other end.
Try it if: you sell something complex or regulated and your page treats a phone call or demo as the first step.
2. A vaguer button label lifted clicks 620.9%
The change: PriceCharting, a paid price-guide site for collectibles, changed a product-page button from 'Download' with the fee shown next to it to 'Price Guide' with no price.
The result: +620.9% clickthroughs, run by PriceCharting on VWO, about one week at 99.9% significance.
The metric is clicks, not purchases. In a follow-up, a lower "Starts at" price lost: $4 beat $2 and $1, though at 91% significance there was no clear winner.
Why it likely worked: 'Download' with a price reads like a commitment. 'Price Guide' describes what people came for, so they click to look before they decide to pay.
Try it if: your main button names an action (download, buy, submit) instead of the thing the visitor wants.
3. A free third unit cut revenue per visitor 52.38%
@CROguy_The change: A store changed its 2-pack at $240 into 'Buy 2 Get 1 Free' at the same $240. Shoppers got three units instead of two for the same money.
The result: -52.38% revenue per visitor ($2.99 to $1.42), run by Milos.
Conversion rate fell from 1.47% to 0.72% and profit per visitor fell 54.01%.
A/B test; the revenue range was wide, but the conversion drop was clear.
Why it likely worked: A better deal at the same price can make people question the product. 'Free' extra units may also suggest the store is clearing stock, or that a buyer needs three to get any value.
Try it if: you plan to sweeten a bundle.
Test the framing before you give away margin. See our bundle tests.
4. A seven-step form beat a one-page form by 154%
The change: HomEquity Bank split its one-page reverse mortgage inquiry form into seven steps.
The result: +154% conversion rate, run by HomEquity Bank.
A/B test with an audience of Canadians 55 and older; the test design was not detailed.
A later test that cut seven steps to four added +53% conversions.
Why it likely worked: Each step asked one easy question, so the form never looked like a wall of fields.
Once people had answered a few, they were invested in finishing.
Try it if: your form has more than five fields and you can group them into short, logical steps. More on multi-step forms.
5. Removing the FAQ lifted sales 62%, but removing FAQ and screenshots lost
The change: AssessmentDay, which sells practice aptitude tests, ran a multi-arm test on a co-branded landing page: one version without the FAQ, one without the screenshots, and one without both.
The result: +62% sales with the FAQ removed and +56% sales with screenshots removed.
Removing both cut sales about 3%. Run by AssessmentDay; split URL test on affiliate traffic only, sample size not given.
Why it likely worked: Each section was doing some of the same job.
Cutting one shortened the page; cutting both left visitors with too little to go on.
Try it if: your page repeats itself. Cut one section at a time, and never assume two winning cuts will add up.
6. One word in a headline nearly doubled purchases
The change: Movexa added the word "supplement" to its landing-page headline for a joint product. Nothing else changed.
The result: +89.97% purchases, run by Source Patrol at 98% significance. A/B test; no sample size given.
Why it likely worked: The old headline never said what the product was.
Visitors from ads had to guess whether it was a cream, a device or a pill.
Try it if: a stranger could read your headline and still not know what you sell.
7. 'Add To Cart' beat 'Add To Bag' by 81.4% in purchases
Conversion FanaticsThe change: An online store changed its button text from 'Add To Bag' to 'Add To Cart'. Nothing else changed.
The result: +81.4% purchases, with +95.3% checkout page views and +22.4% add to cart, run by Conversion Fanatics.
A/B test; the write-up says it had not reached a final winner yet, and no sample size was given.
Why it likely worked: 'Cart' is the word most shoppers expect.
A familiar label removes a tiny moment of doubt at the most important click.
Try it if: your buttons use brand-flavored words.
Treat this as a lead, since the test was not final. More on CTA copy.
8. Removing express checkout buttons raised revenue per visitor 66.18%
Blend CommerceThe change: Jackson's removed the express checkout buttons from below Add to Cart on product pages.
The result: +66.18% revenue per visitor and +36.85% conversion rate, run by Blend Commerce over three weeks at 93.82% significance.
Why it likely worked: Several payment buttons under the main button split attention.
With one clear path, more shoppers clicked Add to Cart.
Try it if: your product page shows three or more buttons of similar weight. See payment and express checkout tests.
9. A one-page checkout lost 29% of subscriptions
The change: A media company merged its three-step subscription checkout into a single page with accordion sections. The steps stayed the same.
The result: -29% subscriptions, run by MECLABS. A/B test; the team thinks the single page looked like more work.
Why it likely worked: Showing every field at once makes the task look bigger, even when it isn't.
This is the same effect behind the HomEquity form win.
Try it if: you're about to move to a one-page checkout because it sounds simpler. Read one-page checkout first.
10. A shorter sign-up form got more sign-ups and fewer paying members
The change: A paid content site cut its first registration page to name, email and password, changed the button text and cleaned up the layout.
The result: Free sign-ups rose about 16%, but paid memberships fell from 0.97% to 0.72% (33 to 26).
Run by MarketingExperiments; seven-day test from 2004 with bundled changes and very small paid counts.
Why it likely worked: Easier sign-up let in more people who were never going to pay.
The win on the top metric hid a loss on the one that earned money.
Try it if: you judge form tests on sign-ups alone.
Track the paid step as a guardrail metric.
11. A price cut brought 19% more subscribers and 15% less revenue
The change: A reference publisher cut its annual plan from $69.95 to $50.00 and updated the savings line. Nothing else changed.
The result: Paid subscriptions went from 183 to 218, but revenue fell from $12,800.85 to $10,900.00, about 15% lower.
Run by MarketingExperiments over two weeks in 2008 with five price arms; $75 earned the most.
Why it likely worked: Demand was less sensitive to price than the team assumed, so the extra buyers didn't make up for the lower price.
Try it if: you assume cheaper always wins.
Test prices above your current one too. See price testing.
12. A trial of the paid plan beat a free-forever plan by 268%
The change: Acuity Scheduling sent its main sign-up button to a 14-day full-feature trial of the $10 a month Professional plan, where users entered card details to keep using the software, instead of a limited free-forever account.
The result: +268% paid sign-ups, run by Acuity Scheduling on VWO with a 99.99% chance to beat the original. Total free sign-ups were not reported.
Why it likely worked: A free-forever plan lets people stay free.
A full trial shows the paid product at its best and puts a clear decision at the end.
Try it if: your free plan is generous and few people upgrade. Read free trial strategy.
13. Sending app users to a web checkout cut sign-ups from 27.0% to 18.1%
The change: Dipsea rebuilt its iOS paywall so every subscribe button sent users to a web checkout, with no in-app purchase option.
The goal was to avoid app store fees.
The result: Initial conversion fell from 27.0% to 18.1% and paid conversion from 6.3% to 5.3%, published by RevenueCat.
About 3,100 new US iOS users per variant over about three weeks.
Adding a 30%-off web option next to in-app purchase also lost.
Why it likely worked: Leaving the app to type card details is a big step on a phone.
Saving the fee did not cover the lost buyers.
Try it if: you're weighing web checkout for an app.
Keep in-app purchase as the control. See paywall tests.
14. A hard paywall got more trials and lost money
The change: BoldVoice removed the close button from its onboarding paywall, so users had to start a trial to continue.
The result: More trial starts, but refunds rose by more than the gain, run by BoldVoice.
A/B test; no figures were shared. The app now judges tests on net revenue per exposed user.
Why it likely worked: Forcing a trial pulled in people who didn't want it. They asked for their money back.
Try it if: you're testing a paywall on trial starts.
Wait for refunds and renewals before you call it. See hard vs soft paywalls.
15. Half as many paywall prompts lifted the main offer 81%
The change: Wizz cut its monetization prompts by about half and showed the rest at moments when users were more open to them.
The result: +81% on the primary offer flow, with LTV up about 20%, run by Wizz.
A/B test retold in a RevenueCat post; no sample size given.
Why it likely worked: Constant prompts teach people to tap them away. Fewer, better-timed ones get read.
Try it if: your app shows an upsell on most sessions. More on paywall timing.
16. A plain page beat the branded design by 51%
The change: A regional marketing commission tested its plain sign-up page for a free events guide against a version with brand fonts, images and colors.
The result: The plain control converted 51% higher, run by MarketingExperiments.
A/B test; no sample size or confidence shared.
A web clinic audience mostly expected the branded page to win.
Why it likely worked: The extra design pulled the eye away from the form and the offer.
On a simple sign-up page, the offer is the design.
Try it if: a redesign is on your roadmap because the page 'looks dated'.
Test the look on its own before you rebuild.
17. Making a button's text 2px bigger lifted clicks 42%
The change: FENTY increased the text on its homepage collection button from 13px to 15px.
The result: +42% CTA clicks, run by FENTY on VWO.
Click metric only; no sample size or significance given.
Why it likely worked: Small text on a busy homepage is easy to skip.
A modest size bump made the button read as a button.
Try it if: your CTA text is under 14px on mobile. Measure sales, not only clicks.
18. A static image beat the homepage video by 52.67% in revenue per visitor
The change: A store replaced its autoplaying homepage hero video with three static image heroes, each with a different angle.
The result: +52.67% revenue per visitor ($1.46 to $2.23) for the best image, run by Dave Diederen.
All three images beat the video. The test ran eight days and every range included zero, so it needs a rerun.
Why it likely worked: Video looks premium but loads slowly and says little in the first second.
An image with a clear message gets to the point.
Try it if: your hero is a video and you've never tested it against a still.
19. A $9.49 shipping fee beat $9 and the original
The change: A store tested three shipping fees against its existing one: $9, $9.49 and $10.
The result: +43.81% revenue per visitor ($3.41 to $4.90) at $9.49, run by Dave Diederen.
The $9 fee lost to control. The range ran from -29% to +192%, so treat it as a lead.
Why it likely worked: Odd: the cheaper $9 fee did worse than $9.49. With conversion near 1%, much of this may be noise, which is a lesson in itself.
Try it if: shipping fees have never been tested on your site.
Test several levels, and plan a rerun. Read about the winner's curse.
20. More clicks, 16.3% fewer sales after a header and footer redesign
Clean CommitThe change: A store decluttered its header and built a fuller footer linking to every sports landing page.
The result: -16.3% conversion rate and -16.5% revenue per visitor while clickthrough rose 12%, run by Clean Commit.
36,149 visitors over 35 days; not significant, with the original at 91.0% probability of being better.
Why it likely worked: More navigation clicks can mean people are hunting, not shopping.
Clicks went up because finding things got harder or more tempting, not because buying got easier.
Try it if: you measure navigation tests on clicks.
Use revenue per visitor as the main metric.
What the winners have in common
Clarity beats cleverness
The one-word Movexa headline, PriceCharting's 'Price Guide' button and 'Add To Cart' all won by saying plainly what the visitor gets or does. If people have to guess, some leave.
Perceived effort matters more than actual effort
The seven-step HomEquity form and the three-step checkout both beat single pages holding the same fields.
What people see at first glance sets how hard the task feels.
The top metric can lie
A shorter form, a price cut and a hard paywall all won on volume and lost on money.
A header redesign raised clicks and cut sales.
Pick the metric that pays the bills before the test starts.
Generosity can backfire
A free third unit, a lower price and a generous free plan all lost to the less generous version.
A deal that looks too good can make people doubt the product, or attract people who won't pay.
Small tests need reruns
The shipping fee and hero video tests had wide ranges that included losses.
Surprising results are more likely to be noise, so confirm them before you roll them out everywhere.
Our win rate research shows how rare real wins are.
What to test next
- Rewrite your headline so it names the product type in plain words.
- Split your longest form into short steps and compare it with the single page.
- Strip your product page to one main button and see if revenue per visitor moves.
- Test a price above your current one, not only below it.
- Run your next bundle or discount test on profit per visitor, not conversion rate.
- Test your hero video against a still image with a clear message.
For the biggest positive results in the archive, see our biggest A/B test wins.
Real results
MarketingExperimentsMarketingExperiments
RevenueCatWatch out for
Copying the result, not the reason. A long page won for Medicare because the product is confusing.
It won't help a $20 impulse buy. Copy the reasoning and test it on your traffic.
Thin evidence. Several tests here have no sample size or ran for about a week.
A big percentage on a small number of orders can vanish on a rerun; read about statistical significance before you trust one.
Calling tests on the wrong metric. Clicks, sign-ups and trial starts are easy to move.
Check revenue, refunds and paid conversion before you ship.
If you want help finding tests like these on your own site, a practitioner can set up the research and the test plan.