112 A/B tests that lost, and what they teach

LiftArchive has 112 A/B tests that lost, shared by 44 practitioners and companies.

Losing tests are rarer in public than in real life, which makes each one worth more: it shows a change that looked like a good idea and still hurt the numbers.

Why losing tests matter

Most teams lose more tests than they win, so knowing what tends to lose saves you a test slot.

A loss also protects you: without the test, the losing change would have shipped and quietly cost money.

Patterns: what tends to lose

Ecommerce elements with the highest share of losing tests (at least 10 tests each):

Where the losses happened:

How to read a loss

Check the sample first: a small test can “lose” by chance.

Look for a novelty effect: some changes dip at first and recover once regulars get used to them.

Split by segment: a change can lose on mobile and win on desktop, or the other way round.

Then decide whether to retest a better version of the idea or drop it.

Every losing test in the database is listed below the article, with filters.

Keep reading

• Ecommerce A/B Test Examples with Real Results
• A/B testing research
• Review & Star Rating A/B Tests
• Bundle & Quantity Break A/B Tests
• Product Page A/B Tests
• A/B Test Sample Size Calculator
• How we collect and credit tests