Module 7: A/B testing · Lesson 7.4 · about 20 min
The peeking problem
Checking an A/B test every day and stopping the first time it looks significant turns a 5 % false positive rate into 20 % or more. Why it happens, and the honest options: fixed horizons, sequential tests and Bayesian monitoring.
Module 7 · about 20 minutes · no new maths. You’ll be able to explain why stopping on a peek inflates false positives, and choose an honest way to monitor a test.
In everyday terms: flipping a coin and stopping the moment heads are “ahead by a lot”. Keep checking and sooner or later they will be, even with a fair coin. Stopping then proves nothing.
1. Each look is another lottery ticket
A p-value threshold of 0.05 assumes you test once, at a sample size fixed in advance. During a test, the running p-value wanders up and down. If you check every day and stop the first time it dips below 0.05, you give noise many chances to cross the line.
Run 500 A/A tests. Checked once at the end, about 5 % “win”, as promised. Stopped at the first daily p < 0.05, around a quarter “win”, all of them false. The lines show why: early on, with little data, the p-value swings wildly.
The same problem in reverse: extending a test because p = 0.07 “is nearly there” and stopping as soon as it crosses 0.05.
2. Honest options
1. Fixed horizon (the default). Plan the sample size (Lesson 7.3), look at the dashboard for bugs and guardrails as often as you like, but make the decision only at the end. Simple and correct.
2. Group sequential designs. Plan a few interim looks (say at 25 %, 50 %, 75 %, 100 % of the sample) with stricter thresholds early on. O’Brien-Fleming boundaries use very strict early thresholds (about p < 0.0001 at the first of four looks) and close to 0.05 at the end, so overall α stays at 5 % and you can still stop early for a dramatic effect.
3. Always-valid / sequential tests. Methods such as mSPRT give p-values (or confidence sequences) that stay valid however often you look. Many commercial A/B tools use these. They need somewhat more data for the same power when you don’t stop early.
4. Bayesian monitoring (Lesson 7.7) with a decision rule based on expected loss. It’s less sensitive to peeking in practice, but it isn’t magic: repeatedly stopping on “P(B better) > 95 %” still raises the false win rate for null effects.
Stopping for harm is different: if a guardrail shows clear damage (errors spike, revenue drops sharply), stop. Protecting users matters more than a clean experiment; just don’t then claim the test “proved” anything about the primary metric.
- Fixed-horizon test
- Look once, at the planned size. False-alarm rate α.
- Peeking
- Many looks, stop at the first p < 0.05: false-alarm rate 20 % or more.
- Sequential design
- Planned looks with stricter thresholds early on (for example O'Brien-Fleming), keeping α at 5 % overall.
- Physical meaning
- Early in a test the estimate swings wildly because few users are in. Each look is another chance to catch it at the top of a swing. The p-value assumes one look; with many looks it is too small.
“It was significant on day 5, so the effect is real and we saved time.” On day 5 of 28 the estimate is noisy and the effect often shrinks or vanishes later. Decide in advance when you will look and what threshold each look needs.
Watch it happen, one step at a time:
Maths and principle
Principle: the chance that a random walk crosses a boundary at some point is much higher than the chance it ends beyond it.
Numbers: with equally spaced looks at α = 0.05 each, the overall false positive rate is about 8 % for 2 looks, 11 % for 3, 14 % for 5, 19 % for 10 and approaching 1 (eventually certain) for unlimited looks. A Pocock design keeps the total at 5 % over 5 looks by testing each at about 0.016.
Analogy: asking “is it raining?” every five minutes all week; sooner or later you’ll see a shower, even in a dry week.
3. Worked example
Made-up story. A team runs a 4-week test of a new banner. On day 5, B is ahead with p = 0.03. They stop and ship. Over the next month, conversion is unchanged.
- Day 5 of 28 had only 18 % of the planned sample. The running estimate at that point was very noisy (the simulation shows how wildly it swings).
- With daily peeking, the chance of a false win was around 20 %, not 5 %.
- Had they planned a sequential design with O’Brien-Fleming boundaries, the day-5 look would have needed p < 0.0001 or so, which 0.03 doesn’t meet.
4. Practice
- A test is planned for 14 days. On day 14, p = 0.08. Your manager suggests 3 more days. What do you say?
- Why is looking at guardrails daily fine?
- Name two ways to allow early stopping without inflating false positives.
Answers
- That’s peeking in reverse: extending until it crosses 0.05 inflates false positives. Report the result as it stands (effect and interval); if the decision needs more certainty, plan a fresh, larger test.
- Because stopping for harm doesn’t create a false “win”; it protects users. The primary metric decision still waits.
- A group sequential design with pre-set boundaries (e.g. O’Brien-Fleming), or an always-valid sequential test (e.g. mSPRT / confidence sequences).
Try it on your own data: the bayesian A/B test tool shows how the chance that B beats A moves as data arrives; watch it, but decide at the planned end.
New words: peeking, optional stopping, fixed horizon, group sequential design, O’Brien-Fleming boundary, always-valid inference, confidence sequence.
- Every extra look at an A/B test is another chance for a false win.
- Daily peeking can push the false-positive rate from 5 % to 20 % or more.
- Either look once at the planned size, or use a sequential design with stricter early thresholds.
- Early estimates are noisy and often shrink later.
Check yourself
Sources
- Johari, R., Koomen, P., Pekelis, L. and Walsh, D. (2017), Peeking at A/B Tests: Why it matters, and what to do about it, KDD 2017 https://doi.org/10.1145/3097983.3097992
- O'Brien, P. C. and Fleming, T. R. (1979), A Multiple Testing Procedure for Clinical Trials, Biometrics 35(3), 549-556 https://doi.org/10.2307/2530245
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.