The novelty effect is a temporary lift in an A/B test that happens because a change is new, not because it is better.
Returning users click on, explore or try the unfamiliar version out of curiosity, so the variant looks like a winner early on, and the gap shrinks or disappears once the newness wears off.
Its mirror image is the primacy effect: a change that looks worse at first because people are used to the old version, then improves as they adapt.
Both come from the same cause, users learning over time, and both mean the result in week one may not be the result you get after launch.
The novelty effect in A/B testing
Microsoft's experimentation team described it plainly in a 2012 KDD paper by Ron Kohavi and colleagues: when a new feature appears, "some users will investigate the new feature, click everywhere," which creates a bias "that dies quickly if the feature is not truly useful."
The same paper notes the long-term effect can still end up positive, flat or negative.
A novelty effect shrinks a result; it does not tell you which way the result will settle.
A clean example comes from Microsoft Edge. In "A Dirty Dozen" (Dmitriev et al., KDD 2017), the team ran a four-week test of a one-time prompt asking users to add a site to their new-tab page.
Clicks rose overall, but the daily gap between variant and control fell quickly.
Splitting the data showed a large lift during the visit when users saw the prompt and no statistically significant change on later visits.
What it looks like in real store tests
Store tests rarely publish daily charts, but some do report the pattern.
When Conversion Fanatics showed every product option fully expanded on mobile, the variant led by more than 40% early in the test and then faded to about -4% conversions.
Had the test been stopped during that early lead, the store would have shipped a loser.
The follow-up is just as useful. The team expanded only the key option groups, with easier-to-tap choices, and that version held up at +36.6% mobile conversions and +6.4% mobile revenue per visitor.
The idea had something real in it; the first version's early lead just wasn't the evidence.
Before/after comparisons are even more exposed to early spikes.
After DFTBA's theme migration, reported by Prismfly, conversion rate was +52% in the month after launch versus the month before, but the case study itself describes a +127% orders spike as short-term, and traffic also rose.
A launch month is a poor stand-in for a normal month.
Blend Commerce
Conversion Fanatics
Clean CommitConversion FanaticsMost fading wins are noise, not novelty
This is the part most articles skip. In the same 2012 paper, Kohavi's team wrote that "most cases of suspected Primacy and Novelty effects are not real, but just a statistical artifact."
Early in a test, the sample is small, so the measured effect swings widely.
As data piles up, extreme early readings get pulled toward the true value, and a cumulative chart makes that look like a trend.
They added that they "could not find a single experiment" where a statistically significant result flipped to a significant result in the other direction because of these effects.
More often, a significant positive got more positive over time.
So a shrinking early lead is usually regression to the mean, and the fix is the same either way: don't call a test on its first few days.
That habit also protects you from the winner's curse, where the tests you pick as winners overstate their own effect.
How to check for a real novelty effect
The checks below come from the Microsoft papers cited above.
You need a tool that lets you segment results; most testing platforms do.
- Plot the daily gap, not just the cumulative one. A cumulative chart smooths away the pattern. Daily or weekly deltas show whether the effect is actually declining.
- Split new and returning visitors. New visitors have no old version to compare against, so the Kohavi paper suggests computing the main metric for new users only, since they are not affected by primacy or novelty.
- Split by visit number. The Edge team compared the first exposed visit with later visits. If the lift lives only in the first exposure, it is novelty.
- Look again after the first week. The 2012 paper says the delta "usually stabilizes after a week" and treats one week as the minimum anyway, to cover day-of-week swings. Our guide to how long to run an A/B test covers the full sizing.
Segment splits need care. A gap between new and returning visitors is not proof of novelty by itself.
When Blend Commerce added a search bar for Jackson's, conversion rate rose 9% overall, with +28% for returning visitors and +5% for new ones over two weeks.
Returning shoppers often know what they want, so search may simply suit them better.
Only a lift that shrinks across weeks for the same group points to novelty.
A worked example
Here is a hypothetical two-week test of a new homepage module, with numbers chosen to show the check:
| Segment | Week 1 lift | Week 2 lift | Reading |
|---|---|---|---|
| Returning visitors | +14% | +3% | Shrinking within the same group: likely novelty |
| New visitors | +4% | +4% | Stable: closer to the long-run effect |
| All visitors (cumulative) | +10% | +7% | Looks like a solid win, hides the decay |
The cumulative +7% overstates what the module will do once everyone has seen it.
The new-visitor +4% is the better estimate, and you would check that its confidence interval still clears zero before shipping.
What to do when you find one
- Extend the test rather than reading the early peak. Decide the new end date in advance so you don't drift into peeking at results until one looks good.
- Judge the change on new visitors, or on later visits only.
- For big changes, keep a small holdout group on the old version after launch and compare months later.
- Be skeptical of attention-grabbing changes. The Dirty Dozen paper names pop-up ads and rapidly changing page elements as typical novelty-effect features: they pull clicks without helping people finish what they came to do.
The primacy effect
The term "primacy effect" means two different things, and most people searching for it want the psychology one. This section covers both.
Primacy effect in A/B tests
In experimentation, the primacy effect is a result that starts small or negative and grows as users adapt.
The 2012 Microsoft paper gives the classic case: change a site's navigation and experienced users "may be less efficient until they get used to the new navigation," which favors the control.
The Dirty Dozen paper adds a second kind: systems that improve as they learn, like a recommendation engine that gets better as it gathers data.
The biggest documented example is from Google. Hohnhold, O'Brien and Tang (KDD 2015) measured how users' tendency to click ads changed over months depending on ad quality, which they called ads blindness and sightedness.
The learning had a half-life of about 60 days.
That work fed into a 50% cut in the ad load on Google's mobile search, which they estimated would be net positive for revenue in the long run.
The half-life number shows why short tests miss these effects.
They modeled learning as decaying at a rate of about 0.012 per day; half-life = ln(2) / 0.012, which is roughly 58 days.
A two-week test sees only a small slice of that curve.
A store-level loss that may involve primacy: when Clean Commit tested a single-row header with an image mega-menu, conversion rate fell 22.5%, orders fell 22.7% and revenue per visitor fell 12.3%, over about a week on 10,957 mobile visitors.
Returning shoppers who knew the old menu may have been part of that drop, but the published result doesn't split them out, so we can't say.
With a loss that large, the Kohavi paper's advice applies: extending a clearly negative test rarely pays off.
| Novelty effect | Primacy effect | |
|---|---|---|
| Early result | Looks better than it is | Looks worse than it is |
| Over time | Lift shrinks | Effect grows |
| Typical cause | Curiosity about something new | Habit with the old version; systems that learn |
| Typical changes | New widgets, prompts, pop-ups, visual refreshes | Navigation, layout and workflow changes; personalization |
| Who is affected | Returning users | Returning users |
| Risk | Shipping a change that later goes flat | Killing a change that would have won |
Primacy effect in psychology
In memory research, the primacy effect is the tendency to remember the first items in a list better than the middle ones.
The standard reference is Bennet Murdock's 1962 study of free recall, which found the first and last words in lists were recalled best.
This is one half of the serial position effect, and it has been reproduced many times.
In first impressions, the primacy effect comes from Solomon Asch's 1946 "Forming Impressions of Personality", where people rated a person differently depending on whether positive or negative traits came first.
The replication record here is mixed. A 2014 replication by Nauts and colleagues found no evidence for the popular "warmth matters most" reading of Asch's work.
The FORRT replication atlas lists a 2018 replication that found support for a small primacy effect in impression formation.
What psychology's primacy effect means for your tests
Research shows order affects memory and impressions in lab tasks.
It does not show that putting your best benefit first will lift sales on your page by any set amount.
That is a hypothesis to test, not a result to assume.
The two meanings also explain each other loosely: people anchor on what they met first, including the old version of your site.
That is one reason returning visitors can resist a redesign at first.
You can browse what moving content up or down actually did in real stores across our A/B test database, and see more on time-related traps in the A/B testing guide and our research section.