Restoring an older mobile menu: +5.4% conversion, -13.8% RPV
Test run byClean CommitCRO agency
Commentary byLiftArchiveOur analysis, not theirsRestoring an earlier, category-focused mobile menu without login and account buttons lifted conversion rate 5.4% but cut revenue per visitor 13.8%.
Clean Commit ran this test for an anonymized Shopify store and published it in its A/B test library.
We treat it as inconclusive: neither move reached significance, and another run of the same revert lost.
- Conversion rate: +5.4%, with only a 66.5% probability that the variant was better. Not significant.
- Revenue per visitor: -13.8%, because average order value fell 18.2%. Negative.
- Orders: +3.1% and add-to-cart rate +4.7%, so more shoppers bought, but they spent less.
- Overall: inconclusive. The sales gain and revenue loss point in opposite directions.
The change
The variant restores the older category-first menu and drops the account buttons.
Simplified mockup drawn by LiftArchive from the shared screenshots.
Control
The newer mobile menu, which included login and account buttons.
Variant
The menu went back to its earlier category-focused layout and original item order, with the login and account buttons removed.
Why it likely came out this way
Our reading of the result, not the practitioner's claim.
- A simpler, category-first menu may have helped some shoppers reach products faster, which fits the small rise in add-to-cart rate.
- Removing account links may have made life harder for returning customers, who often place larger orders.
- With about 8,000 visitors, swings this size can be noise, which may explain why a separate run of the same revert went the other way.
Caveats: The test ran for 16 days on 8,086 mobile visitors and no metric reached significance. A separate run of this revert, with tiles, lost 5.7% on conversion rate.
Who ran it
Clean Commit ran this test for a Shopify store (anonymized) and published the results (Clean Commit A/B test library). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Clean Commit →
How to run this test on your store
- Hypothesis
- If we bring back the older category-focused mobile menu and remove distractions, then more shoppers will buy because a familiar menu is simpler to use.
- Primary metric
- Mobile revenue per visitor
- Guardrails
- Mobile conversion rate
- Average order value
- Returning-customer conversion rate
- Who should test it
- Stores weighing a rollback after a navigation redesign. If many of your buyers log in or return often, keep account links visible, since this test hints that removing them can lower order value.
Questions
Should I remove login and account links from my mobile menu?
Be careful. Here it came with an 18.2% drop in average order value, possibly because returning customers lost an easy path.
What if conversion rate and revenue per visitor disagree?
Decide on revenue per visitor or profit, not conversion rate alone.
More orders that are smaller can still mean less money.
Why run the same test twice?
A rerun shows whether a result holds. Here the two runs disagreed, which suggests the true effect is small or unclear.
Similar tests
@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon