Cohort analysis groups users who share a starting event, such as signing up, first purchasing, or entering an A/B test in the same week, and then tracks how each group behaves over the following days, weeks or months.
Because every group is measured from its own day zero, you can see whether newer users retain, repurchase or pay better than older ones instead of mixing everyone into one blended number.
For conversion work, it is the tool that tells you whether a test winner kept paying off after the first order or the first trial start.
This page covers how cohorts work, a worked cohort analysis example, and real tests from our database where the day-zero result and the cohort result disagreed.
What cohort analysis measures
Optimizely's glossary defines it as "a behavioral analytics technique that groups users with shared characteristics over time to identify patterns and trends."
The key word is time. A cohort report has two clocks: when the user joined, and how long it has been since they joined.
That second clock is what separates a cohort from an ordinary segment.
A segment like "mobile users" is a slice of everyone active right now.
A cohort like "users who signed up in the week of September 7" is a fixed group you follow forward, so its numbers only change as its members age.
Two cohort types cover most needs:
- Acquisition cohorts group users by when they first arrived, signed up or bought. They answer "are the people we acquire now better or worse than the people we acquired before?"
- Behavioral cohorts group users by something they did, such as finishing onboarding or buying in their first week. They answer "which early actions go with long-term value?"
For finer slices of active users without the time dimension, see segmentation.
For where people drop out within a single visit or session, use a conversion funnel instead.
Cohort analysis example: reading a retention table
The classic cohort view is a triangle. Each row is a cohort, each column is time since joining, and each cell is the share of that cohort still active.
The numbers below are illustrative, not from a real store, but the layout is the one you will see in GA4, Amplitude, Mixpanel or a spreadsheet.
| Signup week | Users | Week 1 | Week 2 | Week 3 | Week 4 |
|---|---|---|---|---|---|
| Sep 7 | 1,000 | 41% | 30% | 26% | 24% |
| Sep 14 | 1,200 | 43% | 31% | 27% | (not yet) |
| Sep 21 | 900 | 52% | 39% | (not yet) | (not yet) |
| Sep 28 | 1,100 | 50% | (not yet) | (not yet) | (not yet) |
Read it three ways:
- Across a row to see one cohort's decay. The Sep 7 cohort lost most of its drop-off in week 1 (100% to 41%) and then flattened toward about a quarter of users. A curve that flattens means you have a core of retained users; one that keeps falling toward zero means you don't.
- Down a column to compare cohorts at the same age. Week-1 retention went from 41-43% for the first two cohorts to 50-52% for the next two. That is the comparison a blended "weekly active users" chart hides, because it mixes old and new users together.
- Along the diagonal to spot calendar events. If every cohort dips in the same calendar week, regardless of age, the cause is something that happened that week, such as an outage, a holiday or a price change, not the cohorts themselves.
Suppose a new onboarding flow shipped on September 21. The column view suggests it lifted week-1 retention by roughly 9 percentage points (42% to 51%, averaging each pair).
Work out what that means in users: on the 900-user Sep 21 cohort, 52% is 468 users retained in week 1, against about 378 at the earlier 42% rate.
That is a useful signal, but it is still a before/after comparison.
The September 21 cohort may have come from a different ad campaign or a seasonal spike.
To prove the onboarding flow caused the lift, randomize new users into old and new onboarding and compare the two cohorts side by side. The retention rate calculator handles the arithmetic.
Why A/B tests need cohort analysis
Most A/B tests are judged on what happens in the session: conversion rate, revenue per visitor, trial starts.
That is fine for a button color. It breaks down when the change affects what customers do later, such as subscriptions, trials, refunds, plan mix or price.
The fix is to treat each test arm as a cohort.
Everyone exposed to the control is one cohort, everyone exposed to the variant is another, and you follow both forward to the metric you actually care about.
Here is an illustrative paywall test with 10,000 users per arm:
| Arm | Trial starts | Trial-to-paid | Paying users at day 30 |
|---|---|---|---|
| Control | 400 (4.0%) | 50% | 200 |
| Variant | 500 (5.0%) | 36% | 180 |
On day zero, the variant wins with 25% more trial starts.
Thirty days later, it has 10% fewer paying users.
Without a cohort view, you would ship the loser.
Real tests in our database show the same pattern:
- Subscriptions that look like a loss on day zero. Dave Diederen introduced subscriptions with the option pre-checked by default. Upfront, revenue per visitor fell 9.17%, conversion rate fell 2.86% and AOV fell 6.50%, while the subscription share of orders rose 1,295%. Dave's argument is that a test like this has to be judged on rebills, churn and 30/60/90-day LTV, not first-order RPV.
- Fewer immediate sales, more lifetime value. Zumba sent paywall taps to a web checkout instead of in-app purchase. Immediate conversions dropped 25%, but LTV rose 17%. A test read at day zero would have killed it.
- More trial starts, less net revenue. BoldVoice removed the close button from its paywall. Trial starts went up, but refunds rose by more than the gain. BoldVoice now judges tests on net revenue per exposed user on matured cohorts.
- Lower conversion, higher revenue. Teaching.com replaced a free trial with a 30-day money-back guarantee. Conversion was slightly lower, but revenue and LTV were higher. No figures were published.
- A front-end lift that didn't carry through. Slopes offered a one-tap trial card instead of the paywall and got 25% more trial starts, with trial-to-paid about the same. Here the cohort check confirmed the win was real rather than a flood of low-intent trials.
@DaveDiederenRetention itself can be the outcome. Applica reported that ASL Bloom's goal-led onboarding brought +15% trial conversion, +12% ARPU and +18% 30-day retention.
That was a bundle of changes and the source doesn't say whether it was a split test, so treat it as a direction rather than proof.
Cohorts by time in test: catching novelty effects
A second use inside experiments is cohorting by exposure week.
If a variant's lift is large for users who saw it in week 1 and shrinks for later weeks, you are probably looking at a novelty effect or early noise rather than a durable change.
Ruben's authority badge test is a clean example.
After one week it showed +17% conversion rate at 99.6% probability to beat control.
After four weeks the lift was +1.66%, which Ruben called most likely noise.
Splitting results by exposure week, and committing to a full test duration up front, is what stops a week-one spike from becoming a rollout.
Setting up cohort analysis
Every cohort report needs four decisions:
- The entry event. First visit, signup, first purchase, or first exposure to a test. Pick the one your question starts from.
- The return or outcome event. Any visit, a repeat purchase, a renewal, net revenue. For tests, this should be your primary business metric, not a proxy.
- The granularity. Daily for apps with daily habits, weekly for most SaaS and lead-gen, monthly for ecommerce repurchase cycles.
- The calculation. Whether a user counts as retained only if active in that exact period, or in that period and all earlier ones.
In Google Analytics 4, the cohort exploration lets you set inclusion criteria (first touch, any event, any transaction, any conversion or a custom event), a return criterion, and daily, weekly or monthly granularity.
GA4 offers three calculation types: standard (activity in each period only), rolling (active in that period and every previous one) and cumulative (totals across periods).
Weekly cohorts run Sunday to Saturday, not rolling seven-day windows, and the report shows at most 60 cohorts and 15 breakdown values.
Our GA4 for CRO guide covers the rest of the setup.
Subscription apps usually get better cohort data from their billing tool.
RevenueCat's realized LTV per customer chart, for example, cohorts customers by first seen or first purchase date and marks periods as incomplete while new revenue can still arrive.
That flag matters, as the next section explains.
When cohort analysis misleads
Cohort reports are easy to misread. The common traps:
- Comparing immature cohorts. A two-week-old cohort will always show less LTV than a two-month-old one. Compare cohorts at the same age, and only once both have reached it. RevenueCat's docs make the same point: you need to know whether a difference is "simply due to one cohort being less mature than another, or a true underlying performance difference."
- Treating before/after cohorts as proof. Cohorts from different weeks also differ in traffic source, season and promotions. A cohort shift tells you where to look. A randomized test tells you what caused it.
- Tiny cohorts. A cohort of 80 users swings several points on chance alone. Merge daily cohorts into weekly ones, or put a confidence interval on the retention rate before acting on it.
- Too many cuts. Slicing test cohorts by device, country, source and week, then reporting whichever slice won, is a multiple comparisons problem. Decide your cohort cut before the test starts.
- Waiting when you don't need to. If a change only affects in-session behavior, such as a product page layout, holding the decision for a 90-day cohort costs you testing speed for no gain. Use cohort reads where the change plausibly affects later behavior: subscriptions, trials, pricing, discounts, onboarding and refunds.
A practical middle ground is to pick a leading indicator you can read soon, such as trial-to-paid or first rebill, and confirm it against a longer cohort read on a sample of tests.
Tracking a guardrail metric like refund rate alongside the primary metric catches the BoldVoice pattern early.
Where to go next
If your retention curves keep falling, put your own cohort numbers into the subscription LTV calculator.
To find more tests that reported long-term outcomes alongside the headline result, browse the full test database.