Swatch navigation test at The Shade Store: about -82%, stopped
Test run bySurefootCRO agency
Commentary byLiftArchiveOur analysis, not theirsA change to The Shade Store's swatch navigation cut orders of its most popular swatches by roughly 82%, so the test was stopped early.
Surefoot ran this A/B test for the custom window-treatment brand.
Surefoot counted the loss it avoided as part of the program's results.
- Most-popular-swatch orders: about -82%. This is a clear loss.
- The test was stopped early once the drop showed up, so the variant never went live.
- Effects on other lead types were not shared.
Control
The original swatch navigation.
Variant
A changed swatch navigation. The case study does not say what changed.
Why it likely lost
Our reading of the result, not the practitioner's claim.
- Swatch orders are a key step for a custom product. Shoppers want to see and touch fabric before they commit.
- If the new navigation made the popular swatches harder to find, fewer people would order them.
- Without details of the change, these reasons are our guess, not something the source states.
Caveats: The change is not described, and sample size and duration were not shared. The test was stopped early, so the exact size of the drop is rough.
Who ran it
Surefoot ran this test for The Shade Store and published the results (Surefoot case study). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Surefoot →
How to run this test on your store
- Hypothesis
- If we change how shoppers browse swatches, then swatch orders will rise because the right fabrics are easier to find.
- Primary metric
- Swatch orders, split by popular and other swatches
- Guardrails
- Consultation requests
- Total lead submissions
- Swatch page exits
- Who should test it
- Brands that send samples or swatches before a big purchase. Set a stop rule in advance so a test that hurts a high-value lead type can be ended quickly.
Questions
When should I stop an A/B test early?
Stop early when a variant clearly harms a metric you care about and the drop is large.
Agree the stop rule before launch so it is not a judgment call in the moment.
Is a stopped test a wasted test?
No. It tells you what not to ship and prevents a loss you would otherwise have rolled out to everyone.
Similar tests
@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon@oliverkenyon