Module 7: A/B testing · Lesson 7.3 · about 25 min
Sample size and how long to run
Plan an A/B test's size from the baseline rate, the minimum detectable effect, α and power; turn it into days of traffic, round to whole weeks, and decide what to do when the traffic simply is not there.
Module 7 · about 25 minutes · school maths. You’ll be able to compute the users needed for a conversion test, turn that into a duration, and choose what to do when traffic is short.
In everyday terms: spotting a friend in a crowd. A friend in a bright red coat is easy to see from far away; one in grey needs you to get much closer. Small effects are grey coats: you need many more users to see them.
1. Four inputs
- Baseline rate : the current conversion (from recent data).
- Minimum detectable effect (MDE): the smallest lift worth detecting, set by value and cost, not by hope. Absolute (+0.4 points) or relative (+10 %).
- α: usually 0.05, two-sided.
- Power: usually 0.8.
2. The formula
Visitors per group for a two-sided two-proportion test:
where and . A good quick approximation is
with the absolute difference (it’s Lehr’s rule from Lesson 6.4, since the SD of a 0/1 metric is ).
For a mean metric (revenue per user) use with σ from past data.
3. From users to days
- Days = 2n ÷ eligible users per day (for 50/50).
- Round up to whole weeks. Minimum one week; two is safer.
- Count users, not visits, if users are the unit; returning visitors add less new information than new ones.
- Don’t plan to stop early on a good result (Lesson 7.4).
4. When traffic is short
If the test would take months:
- Test bigger changes. A bolder redesign has a bigger plausible effect.
- Use a more sensitive metric closer to the change (add-to-cart instead of purchase), knowing it’s a step removed from value.
- Reduce variance: CUPED, capping outliers, triggering only affected users.
- Accept more risk: α = 0.10 or power 0.7, stated in advance.
- Don’t test: for low-risk, easily reversed changes, sometimes shipping and monitoring is the sensible choice.
- Baseline p
- Today's rate, such as 4 % conversion.
- MDE
- Minimum detectable effect: the smallest lift worth acting on.
- n per group ≈ 16·p(1 − p) ÷ MDE²
- Rule of thumb for 80 % power at α = 5 %.
- Duration
- Total users ÷ eligible users per day, rounded up to whole weeks.
- Physical meaning
- The test must be long enough for the signal (the lift) to stand clear of the noise (the wobble of the rates). Noise shrinks with √n, so to see half the signal you need four times the data.
“We will stop when it looks significant.” That is peeking (Lesson 7.4). Fix the size in advance; if the test is too long to be practical, test a bolder change or accept a larger MDE.
Watch it happen, one step at a time:
Maths and principle
Principle: the test must make the difference you care about stand out from noise: must be about 2.8 standard errors (1.96 + 0.84) for 80 % power at α = 0.05.
Derivation of the quick rule: SE of the difference ≈ . Setting and solving: .
Check: p = 0.10, δ = 0.02: ; exact formula 3,841 (tests/stats.mjs checks this).
5. Worked example
Made-up numbers. Checkout conversion is 4 %. The team cares about +10 % relative (to 4.4 %). 6,000 eligible users a day.
- Exact formula: about 39,500 per group (the simulation gives the exact figure).
- Total 79,000 users ÷ 6,000 a day ≈ 13.2 days → 2 weeks.
- If they wanted +5 % relative (4.2 %): about 154,000 per group, about 52 days → 8 weeks. Probably too long; the team might settle for detecting +7.5 %, or test a bolder change.
6. Practice
- Quick rule: p = 0.2, detect +0.02 absolute. n per group?
- Revenue per user has SD $80. Detect a $2 difference. n per group?
- The plan needs 50,000 per group and you get 4,000 eligible users a day. How many weeks?
Answers
- 16 × 0.16 ÷ 0.0004 = 6,400.
- 16 × 6,400 ÷ 4 = 25,600.
- 100,000 ÷ 4,000 = 25 days → 4 weeks.
Try it on your own data: the sample size calculator does the exact sums for proportions and means, including unequal splits.
New words: baseline rate, minimum detectable effect, power analysis, test duration.
- n per group ≈ 16·p(1 − p) ÷ MDE² (80 % power, α = 5 %).
- Halving the MDE quadruples n.
- Duration = total users ÷ daily traffic, rounded up to whole weeks.
- Fix size and duration before launch.
Check yourself
Sources
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapter 17 (statistics behind experiments)
- van Belle, G. (2008), Statistical Rules of Thumb, 2nd edition, Wiley, Section 2.4 (the 16/Δ² rule)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.