Experience-based onboarding checklist: users 26% likelier to install
Commentary byLiftArchiveOur analysis, not theirsHotjar replaced its generic onboarding checklist with one of three matched to the user's experience, and users who answered the question were 26% more likely to install Hotjar.
Hotjar ran it, and it was published as an Appcues customer story.
This is not a clean split test: the story does not say what the answering users were compared against.
- Install rate: users who answered the question were 26% more likely to install Hotjar.
- The comparison group and method are not stated, so we treat this as a before/after style result rather than a controlled A/B test.
- Hotjar then built the question into its native sign-up flow.
The change
The variant asks about experience, then shows a matching checklist.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
Every new account saw the same generic onboarding checklist.
Variant
New users were asked "Have you used Hotjar before?" and then shown a Beginner, Intermediate or Advanced checklist. The first step in all three was the same: install Hotjar on your site.
Why it likely won
Our reading of the result, not the practitioner's claim.
- Asking about experience may have made the checklist feel relevant, so users trusted the next steps.
- Clearer later steps could have shown users what they'd get once installed, giving them a reason to install.
- Users who choose to answer a question may already be more engaged, which could inflate the lift.
Caveats: Before/after style result: it only covers users who answered the question, and the comparison is not described. Vendor-published, with no sample size, duration or confidence shared.
Who ran it
Hotjar ran this test and published the results (Appcues customer story). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Hotjar →
How to run this test on your site
- Hypothesis
- If we ask new users about their experience level and show a matching onboarding checklist, then more will complete the key setup step because the guidance feels relevant to them.
- Primary metric
- Share of new accounts that complete setup (for example, installing the tracking code)
- Guardrails
- Sign-up completion rate
- Share of users who skip the question
- Week-1 retention
- Who should test it
- SaaS products with a key setup step and users of mixed experience. Randomize all new sign-ups, not just those who answer, or self-selection will skew the result.
Questions
Does personalizing onboarding improve activation?
It can. Hotjar saw users who answered a single experience question install 26% more often, though that result wasn't from a clean split test.
How many onboarding paths should a SaaS product have?
Start small. Hotjar used three checklists based on one question and kept the key first step the same in all of them.