All lessons

Module 7: A/B testing · Lesson 7.3 · about 25 min

Sample size and how long to run

Plan an A/B test's size from the baseline rate, the minimum detectable effect, α and power; turn it into days of traffic, round to whole weeks, and decide what to do when the traffic simply is not there.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 25 minutes · school maths. You’ll be able to compute the users needed for a conversion test, turn that into a duration, and choose what to do when traffic is short.

The smaller the effect you want to catch, the bigger the test: halve the minimum detectable effect and you need four times the users.

In everyday terms: spotting a friend in a crowd. A friend in a bright red coat is easy to see from far away; one in grey needs you to get much closer. Small effects are grey coats: you need many more users to see them.

1. Four inputs

  1. Baseline rate p1: the current conversion (from recent data).
  2. Minimum detectable effect (MDE): the smallest lift worth detecting, set by value and cost, not by hope. Absolute (+0.4 points) or relative (+10 %).
  3. α: usually 0.05, two-sided.
  4. Power: usually 0.8.
Try it: plan a conversion test
new rate to detect
visitors per group
days to run (round up to whole weeks)
Two-sided test at α = 5 %. The curve shows how fast the required traffic grows as the lift you want to detect shrinks: halve the lift and you need about four times the visitors. Low baseline rates need far more traffic. Run whole weeks so weekdays and weekends are both covered.

2. The formula

Visitors per group for a two-sided two-proportion test:

n=(z1−α/22p‾(1−p‾)+z1−βp1(1−p1)+p2(1−p2))2(p2−p1)2

where p2=p1+MDE and p‾=(p1+p2)/2. A good quick approximation is

n≈16p(1−p)δ2

with δ the absolute difference (it’s Lehr’s rule from Lesson 6.4, since the SD of a 0/1 metric is p(1−p)).

For a mean metric (revenue per user) use n≈16σ2/δ2 with σ from past data.

3. From users to days

  • Days = 2n ÷ eligible users per day (for 50/50).
  • Round up to whole weeks. Minimum one week; two is safer.
  • Count users, not visits, if users are the unit; returning visitors add less new information than new ones.
  • Don’t plan to stop early on a good result (Lesson 7.4).

4. When traffic is short

If the test would take months:

  • Test bigger changes. A bolder redesign has a bigger plausible effect.
  • Use a more sensitive metric closer to the change (add-to-cart instead of purchase), knowing it’s a step removed from value.
  • Reduce variance: CUPED, capping outliers, triggering only affected users.
  • Accept more risk: α = 0.10 or power 0.7, stated in advance.
  • Don’t test: for low-risk, easily reversed changes, sometimes shipping and monitoring is the sensible choice.
Baseline p
Today's rate, such as 4 % conversion.
MDE
Minimum detectable effect: the smallest lift worth acting on.
n per group ≈ 16·p(1 − p) ÷ MDE²
Rule of thumb for 80 % power at α = 5 %.
Duration
Total users ÷ eligible users per day, rounded up to whole weeks.
Physical meaning
The test must be long enough for the signal (the lift) to stand clear of the noise (the wobble of the rates). Noise shrinks with √n, so to see half the signal you need four times the data.

“We will stop when it looks significant.” That is peeking (Lesson 7.4). Fix the size in advance; if the test is too long to be practical, test a bolder change or accept a larger MDE.

Watch it happen, one step at a time:

Baseline. 4 % of users buy. The team cares about +10 % relative: 4.0 % → 4.4 %.
Rule of thumb. 16 × 0.04 × 0.96 ÷ 0.004² ≈ 38,400 per group (the exact formula gives about 39,500).
Traffic. 6,000 eligible users a day: 79,000 ÷ 6,000 ≈ 13.2 days.
Round up. 2 full weeks.
Smaller MDE? +5 % relative would need about 4 times as many users: 8 weeks. Probably too long.
Maths and principle

Principle: the test must make the difference you care about stand out from noise: δ must be about 2.8 standard errors (1.96 + 0.84) for 80 % power at α = 0.05.

Derivation of the quick rule: SE of the difference ≈ 2p(1−p)/n. Setting δ=2.8×SE and solving: n=2×2.82p(1−p)/δ2≈15.7p(1−p)/δ2.

Check: p = 0.10, δ = 0.02: 16×0.09/0.0004=3,600; exact formula 3,841 (tests/stats.mjs checks this).

5. Worked example

Made-up numbers. Checkout conversion is 4 %. The team cares about +10 % relative (to 4.4 %). 6,000 eligible users a day.

  • Exact formula: about 39,500 per group (the simulation gives the exact figure).
  • Total 79,000 users ÷ 6,000 a day ≈ 13.2 days → 2 weeks.
  • If they wanted +5 % relative (4.2 %): about 154,000 per group, about 52 days → 8 weeks. Probably too long; the team might settle for detecting +7.5 %, or test a bolder change.

6. Practice

  1. Quick rule: p = 0.2, detect +0.02 absolute. n per group?
  2. Revenue per user has SD $80. Detect a $2 difference. n per group?
  3. The plan needs 50,000 per group and you get 4,000 eligible users a day. How many weeks?
Answers
  1. 16 × 0.16 ÷ 0.0004 = 6,400.
  2. 16 × 6,400 ÷ 4 = 25,600.
  3. 100,000 ÷ 4,000 = 25 days → 4 weeks.

Try it on your own data: the sample size calculator does the exact sums for proportions and means, including unequal splits.

New words: baseline rate, minimum detectable effect, power analysis, test duration.

  • n per group ≈ 16·p(1 − p) ÷ MDE² (80 % power, α = 5 %).
  • Halving the MDE quadruples n.
  • Duration = total users ÷ daily traffic, rounded up to whole weeks.
  • Fix size and duration before launch.

Check yourself

1Baseline conversion 10 %, you want to detect 12 % with 80 % power at α = 0.05 (two-sided). About how many users per group?
2You halve the minimum detectable effect. How does the sample size change?
3The test needs 9 days of traffic. How long should it run?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapter 17 (statistics behind experiments)
  • van Belle, G. (2008), Statistical Rules of Thumb, 2nd edition, Wiley, Section 2.4 (the 16/Δ² rule)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.