Real data on day one of a trial: +16 points in engagement
Commentary byLiftArchiveOur analysis, not theirsGiving trial users real legal clippings on first login, instead of making them wait up to 48 hours, raised engagement with clippings by 16 percentage points (+125% relative).
AURUM ran this test on Astrea, its legal practice software, and it was published as a VWO case study.
It was a back-end change released through a feature-flag rollout.
- Share of trial users engaging with a clipping: +16 percentage points versus the control.
- That is a 125% relative lift.
- It ran as a controlled feature-flag rollout, not a classic page split test.
Control
Legal clippings, the main input users need in Astrea, were only collected after the account was created. They could take up to 48 hours to appear, so users often took up to three days to reach the first useful moment.
Variant
Recent clippings from just before sign-up were pulled in through an external API. They were ready the moment users first logged in.
Why it likely won
Our reading of the result, not the practitioner's claim.
- Users saw real value on first login instead of an empty product.
- Waiting up to two days in a 10-day trial loses momentum, and some users never come back.
- Real data on day one lets users try the core workflow straight away.
Caveats: A back-end change released through a controlled feature-flag rollout. No sample size, duration or confidence level was shared.
Who ran it
AURUM ran this test and published the results (VWO case study). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on AURUM →
How to run this test on your site
- Hypothesis
- If we load real data into new trial accounts before first login, then more users will engage with the core feature because they see value right away instead of waiting.
- Primary metric
- Share of trial users engaging with the core feature in the trial
- Guardrails
- Trial-to-paid conversion
- Error rates and API failures
- Load time on first login
- Who should test it
- SaaS products that look empty until data syncs or builds up, such as monitoring, analytics or legal tools. It won't help much if users can get value right away with their own input.
Questions
How do I reduce time to value in a SaaS trial?
Remove the wait before users see their own real data.
AURUM did this by pulling in recent records at sign-up, so the product wasn't empty on first login.
Should trial accounts start with sample data or real data?
Real data is often more convincing because it is about the user's own work.
AURUM's lift came from real clippings, not demo content.
Can you A/B test a back-end change?
Yes, with feature flags that put users in a group on the server side.
AURUM ran this change as a feature-flag rollout and monitored it as it went out.