Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing by Ron Kohavi, Diane Tang and Ya Xu (Cambridge University Press, 2020) is the most useful single book on A/B testing.
It explains how large tech companies run experiments, why many test results are wrong, and how to pick metrics you can trust.
Read it first, then add one book on statistics, one on usability and one on persuasion.
This list sticks to books that hold up when you compare them with real results.
For each one we say what it is good for, where it falls short, and which tests in our A/B test database back it up or push back on it.
The CRO books worth reading
Most conversion rate optimization (CRO) books fall into three groups.
Some cover how to run experiments, some cover how people use websites, and some cover the psychology of persuasion.
You need at least one from each, because a test program fails if any of those three is weak.
| Book | Authors, year | Best for | Level |
|---|---|---|---|
| Trustworthy Online Controlled Experiments | Kohavi, Tang, Xu, 2020 | Running tests you can trust | Beginner to advanced |
| Statistical Methods in Online A/B Testing | Georgi Georgiev, 2019 | The statistics behind test results | Intermediate |
| Experimentation Works | Stefan Thomke, 2020 | Building a testing culture | Beginner, managers |
| A/B Testing: The Most Powerful Way to Turn Clicks Into Customers | Dan Siroker, Pete Koomen, 2013 | A short first introduction | Beginner |
| Don't Make Me Think, Revisited | Steve Krug, 2014 | Usability and clarity | Beginner |
| Influence, New and Expanded | Robert Cialdini, 2021 | Persuasion principles | Beginner |
| Nudge: The Final Edition | Richard Thaler, Cass Sunstein, 2021 | Defaults and choice design | Beginner |
| Thinking, Fast and Slow | Daniel Kahneman, 2011 | Background on biases (read with care) | Beginner |
| Web Analytics 2.0 | Avinash Kaushik, 2009 | Measurement and analytics thinking | Intermediate |
| Continuous Discovery Habits | Teresa Torres, 2021 | Customer research that feeds tests | Intermediate |
If you only have time for three, read Kohavi, Krug and Cialdini.
That covers testing method, usability and persuasion. If you'd rather learn through structured lessons, our guide to CRO courses compares the paid options, and the CRO guide covers the basics for free.
Why we check books against real tests
Books tend to tell success stories. Real test programs look different.
In Microsoft's internal paper "Online Experimentation at Microsoft", Kohavi and colleagues wrote that "only about 1/3 of ideas improve the metrics they were designed to improve."
Our own archive leans the other way: 1,447 of its 1,724 results (84%) are wins, 165 are flat and 112 lost.
That isn't because CRO practitioners are three times better than Microsoft.
People publish their winners far more often than their losers.
Keep that bias in mind whenever a book, a case study or a database (ours included) makes testing look easy.
The best A/B testing books
These four books teach you how to run experiments.
They overlap less than you'd expect: one is a practical manual, one is about statistics, one is about culture, and one is a quick introduction.
1. Trustworthy Online Controlled Experiments (Kohavi, Tang, Xu)
This is the book people mean when they search for "trustworthy online controlled experiments."
Cambridge University Press published it in March 2020.
The authors draw on experimentation work at Amazon, Google, LinkedIn and Microsoft.
The chapters are split into five parts: introductory topics, selected topics for everyone, alternatives to controlled experiments, building an experimentation platform, and analyzing experiments.
The chapters most CRO teams get the most from are:
- Chapter 3, Twyman's Law and Experimentation Trustworthiness. Twyman's law says that "any figure that looks interesting or different is usually wrong." The chapter covers the checks to run before you believe a surprising result, including sample ratio mismatch.
- Chapter 7, Metrics for Experimentation and the Overall Evaluation Criterion. How to choose one main metric that reflects long-term value, plus guardrail metrics that stop you shipping damage.
- Chapter 8, Institutional Memory and Meta-Analysis. Keeping a record of past tests so you can learn across them, losers included.
- Chapters 17 and 18 on statistics and variance. The advanced part, useful once you're running tests every week.
Here's how Twyman's law applies to a real result.
Peter Christensen removed the "Track Order" link from a store's main navigation and saw +30% revenue per visitor and +24% conversion rate at 98% significance.
They also noted the test was "only 6 days in" and chose to keep it running.
That's the right response: a big lift after a few days is exactly the kind of number the book tells you to double-check.
The metrics chapter matters just as much. Dave Diederen introduced subscriptions with the subscribe option pre-checked.
Revenue per visitor fell 9.17% upfront, while subscriptions' share of orders rose 1,295%.
Whether that test "lost" depends on whether your main metric counts repeat orders.
The book is written to help you make that call before the test starts, not after.
Where it falls short: the examples come from companies with millions of users.
If your store gets a few thousand visitors a month, some of the advice (tiny effects, huge experiment counts) won't apply.
Pair it with our guide to statistical power so you know which effects your traffic can actually detect.
2. Statistical Methods in Online A/B Testing (Georgiev)
Georgi Georgiev, who built the Analytics Toolkit software, published this book in 2019.
It is written for people in CRO and marketing rather than for statisticians, and it focuses on the ways test statistics get misused.
Read it after Kohavi if you want to understand significance, power, one-tailed versus two-tailed tests and stopping rules properly.
It's the wrong first book if you've never run a test, because it assumes you already know the basic workflow.
3. Experimentation Works (Thomke)
Stefan Thomke is a professor at Harvard Business School.
Experimentation Works: The Surprising Power of Business Experiments came out from Harvard Business Review Press in February 2020. It's about the organization around testing: how companies like Booking.com and Microsoft make experiments part of how decisions get made.
Give this one to a manager or founder who keeps asking for redesigns based on opinion.
It won't teach anyone to analyze a test.
4. A/B Testing: The Most Powerful Way to Turn Clicks Into Customers (Siroker, Koomen)
Dan Siroker and Pete Koomen co-founded Optimizely. Wiley published their book in August 2013, and it opens with Siroker's experience as director of analytics on the 2008 Obama campaign.
It's short and easy to read, which makes it a decent first book.
It's also dated. Both the statistics and the tools have moved on since 2013. Read it for the mindset, then get your methods from Kohavi.
Conversion optimization books on users and persuasion
Testing books tell you how to measure a change.
These books help you decide which changes to test.
Treat each idea in them as a hypothesis, not a rule.
5. Don't Make Me Think, Revisited (Krug)
Steve Krug's usability classic is in its third edition, published by New Riders in 2014 with a new chapter on mobile.
Its main idea is that every moment a visitor spends puzzling over a page costs you.
That idea holds up in many tests. Blend Commerce simplified an apparel brand's mobile menu and saw +11.11% revenue per visitor.
But "simpler" doesn't always win. When Clean Commit tested a stripped-back mobile layout, average order value rose 4.0% but conversion rate fell 4.2% and bounce rate rose 14.3%.
And when Dave Diederen removed an email popup to clean up the experience, revenue per visitor fell 20.51%.
6. Influence, New and Expanded (Cialdini)
Robert Cialdini's Influence is the source of the persuasion principles most CRO teams use.
The 2021 Harper Business edition added a seventh principle, unity, alongside reciprocity, commitment and consistency, social proof, liking, authority and scarcity, as Arizona State University's W. P. Carey School reported.
The research in the book comes mostly from lab and field studies, not website A/B tests.
Some of it has replicated well and some less so.
On real stores, social proof often helps: Dave Diederen added a star rating and review count above the product title and got +8.14% conversion rate and +8.45% revenue per visitor.
Scarcity is less reliable: Clean Commit's scarcity message in the cart showed +2.7% conversion rate, but the result wasn't significant.
Our Cialdini principles guide collects the tests for each principle.
7. Nudge: The Final Edition (Thaler, Sunstein)
Richard Thaler and Cass Sunstein revised Nudge from cover to cover for the final edition (Penguin, 2021).
Its core idea, choice architecture, is very practical for ecommerce: the default option, the order of choices and the framing all change what people pick.
The pre-checked subscription test above shows the catch.
A default can move behavior a lot (subscription share up 1,295%) while hurting the metric you report this month.
Read the book alongside our page on the default effect.
8. Thinking, Fast and Slow (Kahneman)
Daniel Kahneman's 2011 book is the most popular introduction to cognitive biases.
Read it, but be careful with chapter 4 on priming.
A 2017 analysis by Ulrich Schimmack, Moritz Heene and Kamini Kesavan, "Reconstruction of a Train Wreck", found that the priming studies the chapter cites were badly underpowered.
Kahneman replied on that page: "I placed too much faith in underpowered studies," and wrote that priming effects "cannot be as large and as robust as my chapter suggested."
That is a useful lesson for anyone reading test results based on small samples.
9. Web Analytics 2.0 (Kaushik)
Avinash Kaushik's Web Analytics 2.0 (Sybex, 2009) predates GA4 by years, so skip the tool screenshots.
It's still worth reading for how it combines numbers with qualitative data like surveys and usability tests.
That habit is what produces good test ideas.
10. Continuous Discovery Habits (Torres)
Teresa Torres wrote this for product teams, and it came out in May 2021.
Its opportunity solution tree, weekly customer interviews and assumption testing map neatly onto CRO research.
It's especially useful for SaaS and app teams.
A reading order that works
- Start with Krug and Cialdini for ideas worth testing.
- Read Kohavi, Tang and Xu before you trust your next result.
- Add Georgiev when you need to defend your statistics.
- Give Thomke to whoever approves your roadmap.
Then compare each idea with real results. The tests below come from this page.
Each one either backs up a book's advice or shows where it breaks down.
@DaveDiederen
Blend Commerce
@peterac_@DaveDiederen@DaveDiederen
Clean Commit