AI personalization, sometimes sold as hyper-personalization, uses machine learning on real-time behavior to choose what each visitor sees, instead of showing everyone the same page or sorting people into a few hand-built segments.
In our database, it works best for narrow jobs like product recommendations and site search.
Bigger claims, such as an AI that redesigns your whole homepage for each person, rarely come with clean A/B test proof.
This page sorts the two. It starts with what the term means, then goes through the tests in the LiftArchive database that put AI-driven experiences against a control, including the ones that didn't prove anything.
AI personalization and hyper-personalization: what the terms mean
Optimizely's glossary defines hyper-personalization as using "real-time data, AI, and machine learning to deliver highly individualized content and experiences to each customer."
In the same entry, it separates this from basic personalization (name, location) and says it goes "further than segmentation."
In practice, vendors use "AI personalization" and "hyper-personalization" for the same thing.
The useful question is not which label a tool carries but who makes the decision about what to show: a person writing rules, or a model learning from data.
Rules, segments and models
| Approach | Who decides what each visitor sees | Typical example | Main weakness |
|---|---|---|---|
| Rule-based | A marketer writes if/then rules | Show local currency to visitors from the UK | Only as good as the rules you think of |
| Segment-based | A marketer defines groups, then designs a page for each | One homepage for new visitors, another for returning ones | Each segment is smaller, so tests take longer |
| AI / 1:1 | A model picks content per visitor from behavior and context | A "Suggested for you" carousel ranked per session | Hard to see why it chose what it chose; needs lots of traffic |
Most of what works well on websites today sits in the first two rows, or in the third row for one specific slot such as a recommendation widget.
Fully generated, per-visitor pages are the part of the pitch with the least evidence behind it.
What the AI actually does
Under the hood, most AI personalization is a ranking or selection problem.
The model has a pool of options (products, banners, articles, offers) and picks the one it predicts this visitor is most likely to click or buy, then learns from what happens.
A well-known version is the contextual bandit. In a 2010 paper, Lihong Li and colleagues used one to pick news stories on the Yahoo! front page and reported a 12.5% click lift over a standard context-free bandit, tested on more than 33 million events.
Note what that compares: one algorithm against another, not AI against a human-built page.
If you want the mechanics, see our page on multi-armed bandits.
Netflix is the other standard example. In a 2015 paper in ACM Transactions on Management Information Systems, Carlos Gomez-Uribe and Neil Hunt wrote that the recommender system influences choice for about 80% of hours streamed.
They also described judging algorithm changes with A/B tests on retention and engagement, not on offline accuracy alone.
What surveys say, and what they don't
McKinsey's Next in Personalization 2021 report (November 12, 2021) found that 71% of consumers expect personalized interactions and 76% get frustrated when they don't get them.
It also states that personalization "most often drives 10 to 15 percent revenue lift," with company-specific lift spanning 5 to 25%.
Those are survey results and consulting estimates across all channels, not controlled tests of one website feature.
They tell you shoppers like relevance. They don't tell you what a given AI widget will do on your store.
AI website personalization: what the A/B tests show
We pulled every result in the database where an algorithm, rather than a person, chose content per visitor, plus the segment-based tests that are usually sold under the same label.
The pattern is consistent: the strongest results are narrow and measurable, and the broadest ones are often unproven.
Where it works: recommendations
Recommendations are the most tested form of AI personalization, and the results are mostly positive.
For more on placement and format, see our product recommendations guide.
- On Kurt Geiger's homepage, AdaptiveCX added a "Suggested For You" carousel filled from each visitor's in-session category interest. Homepage-to-product-page conversion rose 23.28%. No revenue figures were published.
- Optimizely reports that an online retailer's personalized homepage recommendations beat static ones by 136% on clicks to product pages. That number comes from Optimizely's overview article, and the detail page gives no figure.
- For Branch Furniture, Visually showed a five-product recommendation slider to visitors from the last 60 days. Visually reports +8.9% conversion rate and +21.2% revenue per session, but the test was called within a few days.
Notice the metrics. Two of the three measure clicks, not purchases.
A recommendation carousel almost always gets clicks, so the business question is whether those clicks turn into revenue.
Where it works: one engine against another
Some of the cleanest tests in the database keep the design fixed and swap only the algorithm behind it. That isolates the "AI" part.
- Blend Commerce kept Stone Creek Coffee's ten product-page recommendations in the same spot and swapped Shopify's native engine for Rebuy's. Conversion rose 23%, revenue per visitor 25% and subscription revenue per visitor 84%. The caveat: 3,452 visitors over 4 days, an 89.5% probability to beat control, and the test ended early.
- The Bee's Knees British Imports moved its search and collection pages from Boost AI to Nimstrata's Google Vertex AI-powered search, run by 36 with Nimstrata. Conversion rose 20.6% over five weeks at 99% confidence.
- Clare V. switched cart recommendations to Visually's tool and Visually reports +11.7% conversion rate, with no sample size given.
Two of these are vendor-versus-vendor tests published by the winning vendor.
Treat them as evidence that the engine can matter, not as a forecast for your store.
Blend Commerce
Clean CommitClean CommitBlend CommerceWhere the evidence is weak: rollouts and popups
Many headline AI personalization numbers come from before-and-after comparisons, not randomized tests. Two examples from vendor case studies:
- AB Tasty reports that Bricomarché's move from manually managed to AI-driven recommendations on its homepage, product pages and 404 pages came with +21% homepage conversion and +21% average basket on product pages. The test setup isn't described.
- OptiMonk reports +27% purchases for Gelpro Australia after adding an AI recommendation popup shown after 15 seconds on a product page. The comparison basis isn't stated.
Before-after numbers mix the change with seasonality, promotions and traffic shifts.
They can be right, but you can't tell from the number alone.
Where it didn't prove out: tailored layouts
The closest thing in our database to the "different site for each visitor" pitch is a set of Clean Commit tests on Shopify stores that served different layouts to new and returning visitors. None reached a reliable result.
- A homepage split by new vs returning visitors showed +241.1% conversion rate, +83.7% add-to-cart and +106% AOV, but it didn't reach significance, and overlapping tests and holiday promotions interfered.
- A similar educational homepage for new visitors came out at +0.5% conversion rate and +6.7% RPV, with only a 58.6% probability on RPV. Mobile was up 17.7% and desktop down 24.4%.
- Product pages tailored to new and returning visitors moved conversion rate by +1.0% with flat orders over 2,581 visitors in 33 days, and add-to-cart fell 10.7%.
The +241% figure is the kind of number that ends up on vendor slides.
It's also the clearest warning here: splitting traffic into segments shrinks each one, and big swings on small, noisy samples are common.
Our page on statistical significance explains why that result counts as inconclusive.
Even a recommendation widget can do nothing. Blend Commerce's test of Rebuy recommendations in Fame's checkout showed +2% conversion and +5% revenue, but revenue per visitor moved -0.16% and only two products were bought through the widget.
Simple targeting often does as well
Several of the bigger personalization wins in the database need no machine learning at all.
Dave Diederen's geo-detection popup, which asks international visitors to confirm country and currency, lifted revenue per visitor 13.01%.
Daily Horoscope's country-tailored paywalls report a 183% higher trial start rate.
These are rules a person wrote in an afternoon.
Before paying for 1:1 AI, check whether a few dynamic content rules based on location, traffic source or behavior would capture most of the gain.
What AI personalization can't do
- Fix a weak offer. It ranks what you already have. If the products, prices or copy are wrong, it will pick the least-bad option faster.
- Work on low traffic. Models need data per visitor type. Stores with a few thousand visitors a month will struggle to train a model and to measure it, as the Clean Commit tests show.
- Prove itself. A vendor dashboard that reports "revenue influenced by recommendations" counts sales that might have happened anyway. Only a randomized comparison tells you the incremental effect.
- Explain its choices. When a 1:1 page underperforms, you can't easily see which part failed, which makes the next test harder to design.
How to test AI personalization properly
- Keep a holdout. Leave a random share of visitors on the non-personalized experience permanently, so you can measure the total effect over time. See holdout groups.
- Change one thing. Swap the engine and keep the design, as in the Stone Creek Coffee test, or test the widget against no widget. Don't launch a new design and a new algorithm together.
- Measure revenue per visitor. Clicks and "recommendation engagement" will almost always rise. Revenue per visitor and orders are what pay for the tool.
- Size the test for the segment. If the experience only targets returning visitors, calculate sample size on returning visitors alone with an A/B test sample size calculator.
- Run past the novelty. A new carousel draws attention for a while. Run at least two full weeks and watch for the novelty effect fading.
If you're comparing vendors, our website personalization tools page covers the options, and the personalization hub collects every related test.