A good A/B test hypothesis names the evidence, the change, the audience and the metric in one sentence: "Because we saw [evidence], we expect [change] for [audience] to increase [metric], measured by [primary metric] over [sample size]." If you can't fill in the "because" part with something you actually observed, you don't have a hypothesis yet. You have a guess.
Below is the format, real examples from our database of ecommerce test results, a side-by-side of weak and strong hypotheses, and a copy-paste template.
The format: because, we expect, measured by
Most hypothesis templates in CRO follow the same three-part shape. Craig Sullivan's widely shared Hypothesis Kit uses a "Because / If / Then" structure, and most testing tools copy some version of it. We use a slightly longer version that forces you to commit to a metric before launch.
Each part has a job:
- Because [evidence]: what you saw in analytics, heatmaps, session recordings, surveys, reviews or support tickets. This is the reason the test exists.
- We expect [change]: the one thing you will change, described clearly enough that a developer could build it.
- For [audience]: which visitors see it. All traffic, mobile only, new visitors, a single market.
- To [direction] [metric]: the behavior you expect to move, and which way.
- Measured by [primary metric]: the single number that decides the result, such as conversion rate (CVR) or revenue per visitor (RPV). Our guide to A/B test metrics covers which one to pick.
Two optional lines make it stronger. A minimum detectable effect says how big a lift you need to care about (see minimum detectable effect). A stop rule says how many visitors or days the test needs, which you can get from our sample size calculator.
Why bother writing it down? Because the hypothesis is what you learn from when a test loses. "Countdown timer lost" teaches you nothing. "Urgency doesn't move European shoppers on this store" is something you can use on the next ten tests.
Examples from the database
The best way to see what a strong hypothesis looks like is to work backward from tests that had a clear reason behind them. Here are real results from our database, rewritten in the hypothesis format. The numbers are the practitioners' own reported results.
Evidence from heatmaps
Dave Diederen changed the above-the-fold USPs to the points heatmaps showed buyers engaging with and saw +6.23% CVR.
Because heatmaps show buyers engage most with [specific selling points] further down the page, we expect moving those points into the above-the-fold USPs for all product page visitors to increase purchases, measured by conversion rate.
The evidence and the change line up exactly. The test asks one question: does showing people what they already look for, sooner, make them buy more?
Evidence from knowing the audience
For a store with an older customer base, Dave Diederen increased the product page font size by 4px and saw +4.17% RPV. On the same theme, updating images across the store to show people matching the older target audience's age returned +6.47% RPV.
Because most of our buyers are over [age] and [evidence, e.g. support tickets or recordings show zooming], we expect a 4px larger product page font for all visitors to make the page easier to read and increase revenue per visitor.
Evidence from customer surveys
Surveys are the richest source of "because" lines, since buyers tell you their doubts in their own words. Kasper Dolk added benefit bullets in the hero that answer survey objections, which Kasper reports as $763,679.14 extra profit per year. Oliver Kenyon placed three survey-based reassurance lines above a collection page grid for $44,890/month.
Because post-purchase surveys show the top three hesitations are [objection 1, 2, 3], we expect benefit bullets that answer those objections in the hero, for all new visitors, to increase purchases, measured by profit per visitor.
Oliver Kenyon also added an objection-handling review block below the CTA on mobile, worth +$63,735/month. Same evidence type, different placement, a narrower audience.
@oliverkenyon@oliverkenyon
@DaveDiederen@DaveDiederen@DaveDiederen
@kasper_dolkWhen the hypothesis is the lesson
A written hypothesis earns its keep most on losing tests. Dave Diederen added a countdown timer to checkout on one store and saw +3.16% RPV. On a different test, a countdown timer for a European market lost, at -35.61% RPV.
Ruben saw the same split when testing countdown timers across several brands: +~10% CVR on a low-priced brand, flat on a high-priced brand, and -9% CVR on another. If each test had said "because our shoppers are price-driven impulse buyers," the pattern would jump out. Urgency seems to help some audiences and hurt others, which is a better lesson than "timers work" or "timers don't." More on that on our countdown timers page.
@DaveDiederen@DaveDiederen@DaveDiederen
@RubendgrfWeak vs strong hypotheses
A weak hypothesis usually fails in one of four ways: no evidence, no clear change, no metric, or too many changes at once. Here is how each one looks next to a fixed version.
| Weak | What's missing | Strong |
|---|---|---|
| "Adding a trust badge will increase conversions." | Evidence and audience | "Because survey answers keep mentioning doubts about results, we expect an expert-endorsement badge near the add-to-cart button for new visitors to increase CVR." |
| "Let's test a new product page." | A single, buildable change | "Because recordings show mobile visitors scroll past the variant dropdown without opening it, we expect visual variant blocks on mobile to increase add-to-cart rate." |
| "A countdown timer will create urgency." | Metric and audience | "Because most of our US buyers order during sales, we expect a countdown timer in checkout for US traffic to increase RPV, judged after the full sample size." |
| "New headline, new images and free shipping will boost sales." | One variable | Split it into three hypotheses and rank them with a prioritization framework. |
The first row is not hypothetical. Ruben added a doctor/expert authority badge and saw +1.66% CVR, noted as flat after 4 weeks, with an early +17% after week 1 that turned out to be noise. Dave Diederen's 'Clinician Recommended' badge was flat too. A strong hypothesis with a sample size and stop rule protects you from calling the week-one spike a win.
The metric line matters just as much. When Dave Diederen pre-checked the subscription option by default, the result was -9.17% RPV upfront and +1,295% subscription share. Whether that is a win depends entirely on which metric the hypothesis named before launch.
Quick checks for any hypothesis:
- Can you point to the data behind the "because"?
- Could someone else build the variant from your description alone?
- Is there exactly one primary metric?
- Would a loss teach you something specific?
- Do you know how many visitors the test needs?
If the answer to any of these is no, fix the hypothesis before you build anything. Our list of common A/B testing mistakes shows what happens when teams skip this step, and our research on losing tests shows how often well-reasoned ideas still lose.
Template
Copy this into your test log, a doc, or your testing tool's notes field. Fill in every line before you start building.
- Because we saw [evidence: data source and what it showed],
- we expect [one specific change]
- for [audience: device, market, new or returning]
- to [increase or decrease] [behavior],
- measured by [one primary metric], with [guardrail metrics] as guardrails.
- We need [sample size per variant] or [number of full weeks] to call it, and we care about a lift of at least [minimum detectable effect].
- If it loses, we will learn [what a loss would tell us].
A filled-in example, based on the heatmap test above:
Because heatmaps show buyers engage most with the "free returns" and "ships in 24h" points lower on the page, we expect moving them into the above-the-fold USPs for all product page visitors to increase purchases, measured by CVR, with AOV as a guardrail. We need the full sample size from the calculator and two full weeks. If it loses, we will learn that these points help only once shoppers are already convinced by the product.
Once you have a stack of hypotheses, rank them before testing, then follow the steps in how to run an A/B test.
Some testing tools store hypotheses alongside the tests themselves. Convert's Compass feature, for example, lets you write hypotheses, score them with the PIE or ICE model on a 1-5 scale, track their status, and turn a finished hypothesis into an experiment, according to Convert's documentation. A spreadsheet works fine too, so this is a convenience rather than a requirement. Our Convert review covers the downsides and who it's not for.