# Native Windows build vs Java build: 44 vs 31 sales

- Outcome: Won
- Page type: Other (SaaS)
- Elements: Product options, Friction
- Date shared: 2009-02
- Test run by: [Patrick McKenzie](https://liftarchive.com/practitioners/patrick-mckenzie/)
- Commentary by: LiftArchive (our analysis, not theirs)
- Original source: https://www.kalzumeus.com/2009/02/24/results-of-native-compilation-vs-java-split-test/
- Page: https://liftarchive.com/test/patio11-bcc-native-jet-vs-java-build/

Giving Windows downloaders a natively compiled build, which didn't need Java installed, produced 44 tracked sales versus 31 for the standard Java build. Patrick McKenzie ran this one-month split test for Bingo Card Creator and published it on the Kalzumeus (patio11) blog. Confidence was about 92%, below the usual 95% bar.

## Result

**44 vs 31 sales**

- Tracked sales: 44 for the native build vs 31 for the Java build, out of 75 sales tied to the test.
- Confidence: about 92% (binomial test, p of about 0.08). Suggestive, not conclusive.
- Of 118 total sales that month, only 75 could be tied to a version, so many buyers were not counted.

## Control

Half of Windows downloaders got the standard Java version, wrapped with Launch4j. It asked users to install Java if they didn't have a recent enough version.

## Variant

The other half got a self-contained native build made with Excelsior JET. It removed the Java dependency but was a much larger download.

## Why it likely won (our reading)

- Removing the Java install prompt cut a step where some users likely gave up before trying the app.
- For non-technical users, a single installer that just works may matter more than a smaller download.
- With 75 sales the gap could partly be chance, so the size of the real lift is unclear.

## Caveats

Small sample over one month with about 92% confidence. This tested the downloadable desktop product itself, not a web page, and the native build had a file-saving bug on Windows Vista at the time.

## How to run this test

- Hypothesis: If we remove a required third-party install from onboarding, then more trial users will reach the product and buy because they skip a step where many drop off.
- Primary metric: Paid conversions per trial start
- Guardrail: Download or onboarding completion rate
- Guardrail: Support tickets
- Guardrail: Crash or error rate
- Who should test it: Software and SaaS products whose setup depends on a plugin, runtime or extra install, especially with non-technical buyers. It matters little for browser-based products with no install step.

---

Credit: Patrick McKenzie ran this test and owns the result. Summary and commentary by LiftArchive (https://liftarchive.com/test/patio11-bcc-native-jet-vs-java-build/).
Usage: Free to use, quote and build on, with credit and a link: name the practitioner or organization that ran the test and link to its LiftArchive page. The results belong to the practitioners who ran and shared them; the summaries and commentary are LiftArchive's.