Native Windows build vs Java build: 44 vs 31 sales
Commentary byLiftArchiveOur analysis, not theirsGiving Windows downloaders a natively compiled build, which didn't need Java installed, produced 44 tracked sales versus 31 for the standard Java build.
Patrick McKenzie ran this one-month split test for Bingo Card Creator and published it on the Kalzumeus (patio11) blog.
Confidence was about 92%, below the usual 95% bar.
- Tracked sales: 44 for the native build vs 31 for the Java build, out of 75 sales tied to the test.
- Confidence: about 92% (binomial test, p of about 0.08). Suggestive, not conclusive.
- Of 118 total sales that month, only 75 could be tied to a version, so many buyers were not counted.
Control
Half of Windows downloaders got the standard Java version, wrapped with Launch4j. It asked users to install Java if they didn't have a recent enough version.
Variant
The other half got a self-contained native build made with Excelsior JET. It removed the Java dependency but was a much larger download.
Why it likely won
Our reading of the result, not the practitioner's claim.
- Removing the Java install prompt cut a step where some users likely gave up before trying the app.
- For non-technical users, a single installer that just works may matter more than a smaller download.
- With 75 sales the gap could partly be chance, so the size of the real lift is unclear.
Caveats: Small sample over one month with about 92% confidence. This tested the downloadable desktop product itself, not a web page, and the native build had a file-saving bug on Windows Vista at the time.
Who ran it
Patrick McKenzie ran this test and published the results (Kalzumeus (patio11) blog). The numbers are theirs; the summary and commentary on this page are ours.
Read the original case study on Patrick McKenzie →
How to run this test on your site
- Hypothesis
- If we remove a required third-party install from onboarding, then more trial users will reach the product and buy because they skip a step where many drop off.
- Primary metric
- Paid conversions per trial start
- Guardrails
- Download or onboarding completion rate
- Support tickets
- Crash or error rate
- Who should test it
- Software and SaaS products whose setup depends on a plugin, runtime or extra install, especially with non-technical buyers. It matters little for browser-based products with no install step.
Questions
Do install dependencies hurt software conversion?
They can. In this test the build with no Java requirement drew more sales, though confidence was only about 92%.
Is 92% confidence good enough to act on?
It depends on the cost of being wrong.
The author switched because the downside was small, but a 95% bar is more common.
How can you track sales by version in a desktop app?
Tag links from the app with the version, then credit purchases to that tag.
Buyers who arrive another way won't be counted.