Full site and app redesign: +10% trial sign-ups, purchases flat
Commentary byLiftArchiveOur analysis, not theirsA full redesign of the Bingo Card Creator marketing site, app and purchasing page raised trial sign-ups on AdWords landing pages from 20.9% to 23.1%, but purchases stayed flat.
Patrick McKenzie ran this sitewide A/B test and published it on the Kalzumeus blog in 2012. Many changes shipped together, so no single element can take credit for any part of the result.
- Trial sign-ups on AdWords landing pages rose from 20.9% to 23.1%, about 1,400 visits per version over 7 days. Significant at 90% but not 95%.
- User success, meaning trial users who actually made bingo cards, rose about 15%.
- Purchases were flat: 50 for the old version and 46 for the redesign, a null result.
- Once users reached the new purchasing page, they bought at the same rate as before.
Control
The old design of the marketing site, the app and the purchasing page.
Variant
A new design across the marketing site, the app and the purchasing page and cart. The app also gained a step-by-step progress bar showing how far a user had got in making bingo cards.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- Fewer users hit the 15-card trial limit, which was the main reason people went to buy. Users got what they needed for free.
- The redesign may have drawn in more parents with small needs, who would never pass the trial limit.
- With dozens of changes bundled, any one of them could have nudged users away from the trial limit.
Caveats: This was a bundled redesign with many changes at once, so results can't be tied to one element. Purchase counts were small, and the redesign was adopted anyway for the better user experience.
Who ran it
Patrick McKenzie ran this test and published the results (Kalzumeus (patio11) blog). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Patrick McKenzie →
How to run this test on your site
- Hypothesis
- If we redesign the site and app to be cleaner and show progress through setup, then more visitors will start and succeed with a trial because the path is clearer, which should lead to more purchases.
- Primary metric
- Paid purchases per visitor
- Guardrails
- Trial sign-up rate
- Share of trial users hitting the plan limit
- User activation rate
- Who should test it
- SaaS products with a free plan or trial limit where a redesign is already planned. Track every funnel step, not just the final sale. If you need to know which change did what, test elements one at a time instead.
Questions
Should you A/B test a full redesign?
It can protect you from shipping a redesign that hurts sales.
But with many changes bundled, you learn whether the whole package works, not which part did it.
Why can more sign-ups lead to no more sales?
Gains early in the funnel can be cancelled out later.
Here, happier trial users hit the free limit less often, so fewer of them needed to pay.
Do progress bars help user onboarding?
In this redesign, user success rose about 15%, but a progress bar shipped with many other changes.
It needs its own test to know how much it helped.