Lower-friction subscription form layout: +13.5% subscriptions
Test run byMarketingExperimentsResearch publisher
Commentary byLiftArchiveOur analysis, not theirsA new layout for an online subscription service's sign-up form raised the subscription rate by 13.5%, once days affected by an Easter promotion were removed from the data.
MarketingExperiments (MECLABS) ran the test with the company, which is not named.
With every day included, the lift was only 6.6% at 62% confidence, which would have looked like no result.
- Excluding Easter promotion days: subscription rate up 13.5%, at 95% confidence.
- Including all days: subscription rate up 6.6%, at only 62% confidence. On that data alone, the test looks inconclusive.
- The company was advised to roll out the new form based on the cleaned data.
The change
Only the subscription form's layout changed.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
The subscription form in its original layout.
Variant
The same form re-laid-out to reduce friction while filling it in. The exact layout changes were not described in the post.
Why it likely won
Our reading of the result, not the practitioner's claim.
- A simpler layout likely made the form feel quicker to finish, which matters most to visitors with normal, not boosted, motivation.
- The Easter email brought in highly motivated visitors who would subscribe on either form, which drowned out the layout effect.
- Removing those days showed how the form performs for everyday traffic.
Caveats: The win only reached 95% confidence after removing promotion days from the analysis, which is a judgement call made after the fact. Sample size and the exact form changes were not shared, and the data is from 2013.
Who ran it
MarketingExperiments ran this test for an online subscription service and published the results (MarketingExperiments). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on MarketingExperiments →
How to run this test on your site
- Hypothesis
- If we re-lay-out our subscription form to cut friction, then more visitors will finish subscribing because the form will feel shorter and easier.
- Primary metric
- Subscription rate from form view
- Guardrails
- Form error or abandonment rate
- Paid conversion or retention of new subscribers
- Results with and without promotion days
- Who should test it
- Subscription SaaS and consumer subscription services with a multi-field sign-up form. Avoid launching it during a big promotion, or plan from the start to analyse promo days separately.
Questions
Should you run A/B tests during a promotion?
It is better to avoid it. Promotions change who visits and how motivated they are, which can hide or exaggerate the effect of what you are testing.
Is it OK to remove promotion days from test data?
It can be, if you knew about the promotion and decided why before looking at the results. Report both versions so readers can judge.
Does form layout really affect sign-ups?
It can. Here a layout change alone was worth 13.5% more subscriptions on normal traffic.