All lessons

Module 7: A/B testing · Lesson 7.8 · about 20 min

A/B testing pitfalls

Novelty effects, the winner's curse, too many metrics and segments, users who interact with each other, and the habit of trusting results that look too good. The checks that catch them.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 20 minutes · no new maths. You’ll be able to spot the common ways a “winning” test misleads, and know the check for each.

Lifts from tests usually shrink after launch: novelty wears off, lucky winners regress, and wins from different tests do not simply add up.

In everyday terms: a new café on the high street is packed in its first week because it is new. Judging its long-term business by that week overestimates it badly. A redesign gets the same first-week bump.

1. Novelty and learning effects

People react to change itself. A shiny new button gets clicks because it’s new (novelty effect); a rearranged menu annoys regulars at first (learning effect, or change aversion). Either way, the first days mislead.

Try it: does the lift last?
estimate if stopped then
true long-run lift
Dots: each day's measured lift (%) for 6 weeks, with noise. Line: the running estimate from day 1. A negative launch value is the opposite, a learning curve: people dislike change at first and come round. Either way, plot the lift by day, compare new and returning users, and run long enough for it to settle. Made-up numbers.

With the starting settings, stopping after one week estimates about +5 %; the true long-run lift is +1 %. Run to six weeks and the running estimate falls to about +2 %, much closer; the last weeks alone sit right on +1 %. Set the launch value negative to see a change that looks bad at first and comes good.

Checks: plot the lift by day; compare new users (who have nothing to be surprised by) with returning ones; run at least two full weeks; for big changes, keep a small holdout group on the old version for months.

2. The winner’s curse

If a test has low power, it only reaches significance when noise pushes the estimate up. So significant results from small tests overstate the effect, sometimes by double. Over a portfolio of tests, the sum of all “wins” is bigger than the real gain.

Checks: plan for power (Lesson 7.3); treat a surprising win as a hypothesis and replicate it; shrink estimates towards zero (a Bayesian prior, Lesson 7.7, does this).

3. Too many metrics and segments

Twenty metrics or twenty segments (“it worked for Android users in Ontario on weekends!”) give about one false win at 5 % each (Lesson 6.8).

Checks: one primary metric decided in advance (Lesson 7.2); treat segment findings as ideas for the next test; if you must look at many, correct for it (Holm or Benjamini-Hochberg).

4. Interference between users

Standard tests assume one user’s group doesn’t affect another’s outcome. That fails when users share something:

  • Marketplaces: a discount for buyers in B uses up stock that buyers in A wanted.
  • Social features: a sharing tool in B sends messages to users in A.
  • Shared budgets: an ad test where both groups bid from the same budget.

Checks: randomise larger units (cities, time windows, clusters of friends) and analyse at that level.

5. Twyman’s law

“Any figure that looks interesting or different is usually wrong.” A +30 % lift from a button colour is almost always a bug: a tracking change, SRM (Lesson 7.6), bots, or a metric that double counts. Check the plumbing before celebrating.

Novelty effect
Users react to anything new; the effect fades.
Winner's curse
Tests that pass with low power overstate the true effect.
Interference
Users in A and B affect each other (marketplaces, social features), so the groups are not separate.
Non-additivity
Two +3 % wins on the same page rarely give +6 %.
Physical meaning
A test measures a short window in an artificial setting. The long-run effect is usually smaller, so trust repeated, well-powered results and holdout groups over single exciting numbers.

“12 winning tests at +1.5 % each mean +18 % this quarter.” Inflated winners, novelty and overlap mean the real total is usually far smaller. A long-running holdout group shows what all the changes really added.

Watch it happen, one step at a time:

The report. 12 winning tests add up to +18 % conversion.
Reality. Conversion over the quarter rose 3 %.
Winner's curse. Several tests had about 30 % power, so their winning lifts were inflated.
Novelty. Three wins were measured in the first week of eye-catching redesigns.
Fix. Power tests properly, run whole weeks, re-test big winners, and keep a holdout group.
Maths and principle

Principle: selecting results on significance biases their size upward.

How big is the curse? If the true effect is δ and the test’s SE is s, a significant estimate must exceed about 1.96s. When δ is well below that (low power), the average significant estimate is close to 2s or more, however small δ is.

Example: true lift 0.5 points, SE 0.5 points (power about 17 %). Significant positive estimates average about 1.25 points, two and a half times the truth.

Novelty: if the lift on day t is L+Ne−t/τ, the average over the first d days is biased upwards by about Nτ/d once d is several times τ, so longer tests shrink the bias.

6. Worked example

Made-up story. A team’s quarterly report lists 12 winning tests adding up to +18 % conversion. Actual conversion over the quarter rose 3 %.

  • Several tests had power around 30 %: their significant lifts were inflated (winner’s curse).
  • Three wins were measured in the first week of eye-catching redesigns (novelty).
  • Two “wins” were in segments picked after the fact.
  • Lifts from different tests on the same page don’t simply add up.

The fix: plan power, run full weeks, one primary metric, replicate surprising wins, and keep a long-term holdout to measure the combined effect.

7. Practice

  1. How would you tell a novelty effect from a real lift?
  2. A test with 15 % power comes out significant with +8 %. What do you expect the true effect to be?
  3. Suggest a randomisation unit for testing a new delivery-fee rule in a food delivery app.
Answers
  1. Plot lift by day (novelty fades), compare new and returning users (new users show no novelty), and use a long-term holdout.
  2. Probably much smaller than 8 %, possibly near zero; replicate before trusting it.
  3. City (or city × week), because riders, restaurants and couriers are shared within a city.

Try it on your own data: the bootstrap test tool lets you see how much an estimate wobbles by chance; sample size tells you whether a test had the power to be trusted.

New words: novelty effect, learning effect, holdout group, winner’s curse, interference, Twyman’s law.

  • Watch for novelty: compare early and late weeks of the test.
  • Low-powered wins exaggerate effects (the winner’s curse).
  • Interference between users breaks the A/B separation; randomise by cluster if needed.
  • Lifts from separate tests do not simply add; use a holdout to measure the total.

Check yourself

1A new design shows +6 % in its first week and +1 % by week six. What is the likely explanation?
2Tests that only just reach significance tend to overstate the true effect. Why?
3A test of a ride-sharing price change randomises riders. What could go wrong?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 3 and 23
  • Gelman, A. and Carlin, J. (2014), Beyond Power Calculations: Assessing Type S and Type M Errors, Perspectives on Psychological Science 9(6), 641-651 https://doi.org/10.1177/1745691614551642

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.