To choose a CRO agency, ask to see real A/B test results from stores like yours, including the tests that lost, and ask how they decide when a test is finished. A good agency will explain its research process, its stopping rules and its contract terms in plain numbers. If they can't show any of that, keep looking.
This guide gives you the questions to ask, the red flags to watch for, the proof to request and the contract terms to check. Every example comes from real tests in the LiftArchive test database, credited to the practitioners who ran them.
Questions to ask a CRO agency
A sales call shows you how an agency talks. These questions show you how it works. Ask them before you see a proposal, so the proposal has to answer them.
1. "Walk me through your last three tests for a store like mine."
You want specifics: the page, the hypothesis, the metric, the sample size, the result and what they did next. A vague answer like "we lifted conversions 30%" tells you nothing about how they got there.
2. "What metric do you call a win on?"
Conversion rate alone can mislead. A discount can raise orders while cutting revenue. Ask whether they judge tests on revenue per visitor (RPV), profit per visitor or conversion rate, and why.
Kasper Dolk at Maison Commerce often reports profit, not just orders. In one testimonial test, the result was +7.22% profit per visitor with a 94.4% probability to beat control. That's the level of detail to expect.
3. "When do you stop a test?"
The right answer names a sample size or duration decided before launch. Statistician Evan Miller explains why: "if you peek at an ongoing experiment ten times, then what you think is 1% significance is actually just 5% significance." Our guide to statistical significance covers the basics.
4. "How do you decide what to test first?"
Look for research before ideas: analytics, session recordings, customer surveys and reviews. An agency that starts with a list of "best practices" is guessing. Ask to see the research notes behind one of their recent tests.
5. "Does a win on another store mean it will win on mine?"
The honest answer is no. The same idea can win, lose or do nothing depending on the store. Dave Diederen saw +3.16% revenue per visitor after adding a countdown timer to checkout. Yet a countdown timer for a European market lost 35.61% RPV, and Kasper Dolk saw CVR +25.61% and RPV +26.18% by removing a checkout countdown timer.
An agency that knows this will treat past wins as ideas to test, not fixes to install.
@DaveDiederen
@kasper_dolk@DaveDiederen
@Rubendgrf6. "Who does the work?"
Ask who writes the hypotheses, who designs the variants and who codes them. Find out if the senior person on the sales call will touch your account after you sign.
7. "What happens to the winners?"
A winning test is only worth money once it ships. Ask whether they hard-code winners into your theme or leave them running inside the testing tool, and who pays for that development.
Red flags when hiring a CRO agency
Some warning signs show up in the first call. Others appear only when you read the case studies closely.
- Guaranteed lifts. No one can promise a result before testing. Many tests lose or come out flat, which is why we track losing A/B tests as closely as wins.
- Only wins in the portfolio. An agency that has run many tests has plenty of losers. Hiding them suggests it doesn't learn from them, or doesn't run many tests.
- Calling tests after a few days. Ruben at Tailored Commerce saw an authority badge test at +17% after week 1. After four weeks it settled at +1.66% CVR, effectively flat. The early number was noise.
- The same checklist for every store. If the audit reads like it could apply to any store, it probably does.
- No talk of traffic. A store with little traffic can't reach a reliable result on small changes in a reasonable time. A good agency will tell you this and suggest bigger changes or other methods. See A/B testing with low traffic.
- Asking for your login. On Shopify, a partner should request a collaborator account. Shopify says collaborators don't count towards your store's user limit, you set their permissions and you can remove them at any time.
Proof to request: real test results
Case studies are marketing. Test results are evidence. Ask for at least three full test write-ups, ideally from stores in your category and price range.
What a complete test result includes
- The page and the exact change, with screenshots of control and variant
- The hypothesis and the research behind it
- The primary metric (RPV, profit per visitor or conversion rate)
- Sample size, run time and the confidence or probability reported
- How any dollar figure was estimated (measured during the test, or projected forward)
Ask for losses, too
A loss write-up shows you how the agency thinks. Milos, who works with stores doing $200k to $1M a month, published a test where changing a 2-pack offer to "Buy 2 Get 1 Free" lost 52.38% RPV, an estimated -$104,679 a month. That kind of result is worth finding in a test, before it ships to everyone.
Dave Diederen shares flat results in public as well, such as moving the review rating above the product title, which showed no CVR change.
Check the confidence behind big numbers
Large revenue figures need context. Kasper Dolk reports a confidence level next to each one, such as a navigation test at +€127,112 revenue lift at 90% confidence. Ask what threshold the agency uses and whether a result at 89% confidence still "in progress" counts as a win in their reporting.
@kasper_dolk
@peterac_@kasper_dolk
@CROguy_@DaveDiederen@RubendgrfLook for volume
Experience across many tests helps an agency spot what is likely to matter. Peter Christensen's Scaling CRO says it has run more than 1,000 A/B tests for 8-figure ecommerce brands. One of Peter's tests, removing the "Track Order" link from the main navigation, produced +30% RPV. Small, specific changes like that tend to come from research, not templates.
You can browse results from every practitioner in our database in our practitioner profiles and compare their work before you get in touch.
Contracts and minimums
Most CRO agencies work on a monthly retainer, with some offering one-off audits or per-test pricing. For typical ranges, see our CRO pricing guide. Whatever the model, read the contract for these points.
Minimum term and notice period
Many agencies ask for a minimum term because research and the first round of tests take time. That is reasonable. Make sure you know the notice period and what happens to tests that are still running when you leave.
Traffic and revenue minimums
Some agencies only take stores above a set revenue or traffic level. Treat that as a good sign: it means they won't sell you a testing program your traffic can't support. If you are below the line, a CRO audit or a solo consultant may be a better first step. Our list of consultants includes smaller options.
Testing tool costs
Ask whether the testing tool license is included or billed to you, and whose account it sits in. If the account is yours, you keep the test history when the contract ends.
Ownership of work
Confirm that you own the designs, code, research and test reports. Ask for a shared document or dashboard with every test, not just a monthly slide deck.
Reporting cadence
Agree on how often you get results and in what form. A short written update after each test ends is more useful than a monthly call that covers everything at once. Ask that each update include the metric, sample size and confidence, the same details you asked for during the sales process.
Performance pricing
Some agencies tie fees to results. Before signing, agree in writing on how lift is measured, over what period, and whether projected revenue counts. Without that, the bill can rest on a number neither side can check.
Access and offboarding
Give access through a collaborator account and grant only the permissions the agency needs. Shopify lets store owners require a collaborator request code, so only partners you share it with can ask for access. When the contract ends, remove the account and check that leftover test scripts are gone from your theme.
Where to start
Shortlist two or three agencies, send each the same questions from this guide and compare the answers side by side. The CRO agencies and experts directory lists practitioners whose real test results you can check before the first call.