Module 6: Hypothesis tests · Lesson 6.8 · about 20 min
Many tests at once: the multiple comparisons trap
Test enough segments, metrics or variants and something will look significant by luck. The family-wise error rate, Bonferroni and Holm corrections, the false discovery rate, and habits that keep dashboards and A/B readouts honest.
Module 6 · about 20 minutes · school maths. You’ll be able to explain why many tests produce false positives, apply Bonferroni and Holm corrections, and set up analyses that avoid the trap.
In everyday terms: throw 20 darts blindfolded at a board and one will probably hit something. Testing 20 segments at 5 % is the same: one “significant” segment is expected by chance alone.
1. Slice it enough and something wins
A test at α = 0.05 has a 5 % false alarm rate. Run 20 independent tests where nothing is going on and the chance that at least one comes out significant is
That happens all the time in analytics: a test that “didn’t win overall” is sliced by device, region, age, new vs returning, day of week… until one slice wins. Or a dashboard tracks 30 KPIs and flags whichever moved “significantly” this week.
Every segment here is pure noise. Run a few analyses: with 20 segments, one or two bars usually cross the 0.05 line. After the Holm correction, almost none do.
2. Corrections
The family-wise error rate (FWER) is the chance of at least one false positive across a family of tests. To keep it at 5 %:
- Bonferroni: test each of p-values against (equivalently, multiply each p by ). Simple and safe, a bit strict.
- Holm: sort the p-values from smallest; compare the smallest with , the next with , and so on, stopping at the first that fails. Always at least as powerful as Bonferroni, with the same guarantee. Use it by default.
When you’re screening many things and can tolerate some false leads (which 50 products to look at more closely), control the false discovery rate (FDR) instead: the expected share of your “discoveries” that are false. The Benjamini-Hochberg procedure does this and is far less strict than Bonferroni.
3. Habits that help more than formulas
- Decide before you look: one primary metric, a short list of secondary metrics and segments, written down before the test starts.
- Label exploration as exploration: “In an unplanned look at 12 segments, mobile users in Quebec showed a lift (unadjusted p = 0.03). Worth a follow-up test.”
- Replicate: a surprising segment win becomes the hypothesis of the next test, not a shipped decision.
- Count the forking paths: trying several date ranges, outlier rules or metric definitions is also multiple testing, even if you only report one.
- Family-wise error rate
- Chance of at least one false alarm across all the tests: 1 − 0.95^k for k independent tests.
- Bonferroni
- Test each at α ÷ k. Simple, strict.
- Holm
- Sort p-values; compare the smallest with α/k, the next with α/(k − 1), and so on; stop at the first failure. Always at least as powerful as Bonferroni.
- False discovery rate
- Controls the share of "wins" that are false rather than the chance of any false win.
- Physical meaning
- Significance is a ticket price. If you buy many tickets, you must pay more per ticket, or the few winners you find will mostly be luck.
“Each segment was tested at 5 %, so each result is 95 % solid.” With 10 segments, the chance of at least one false win is 1 − 0.95¹⁰ ≈ 40 %. Pre-register a few segments and correct for them.
Watch it happen, one step at a time:
Maths and principle
Principle: each test has a small chance of a false alarm; across many tests those chances pile up.
Equations: FWER for independent null tests at level α: . Bonferroni threshold . Holm: for sorted , reject while . Benjamini-Hochberg: find the largest with and reject the first .
In code: Python statsmodels.stats.multitest.multipletests(pvals, method="holm") (or "bonferroni", "fdr_bh").
4. Worked example
Made-up numbers. A test readout checks 5 pre-planned segments. p-values: new users 0.004, returning 0.03, mobile 0.04, desktop 0.20, tablet 0.65.
- Unadjusted, 3 look significant.
- Bonferroni (threshold 0.05 ÷ 5 = 0.01): only new users (0.004).
- Holm: sorted 0.004 vs 0.05/5 = 0.010 ✓; 0.03 vs 0.05/4 = 0.0125 ✗, stop. Only new users.
- Holm-adjusted p-values: 0.004 × 5 = 0.02; 0.03 × 4 = 0.12; 0.04 × 3 = 0.12; then 0.40 and 0.65.
Report: “The lift was clear for new users (adjusted p = 0.02). Returning and mobile users showed smaller lifts that don’t survive correction for testing 5 segments.”
5. Practice
- 10 tests, nothing real. Chance of at least one false positive at 0.05?
- Bonferroni threshold for 8 tests at α = 0.05?
- Holm on p = 0.01, 0.02, 0.03 (3 tests). Which are significant at 5 %?
Answers
- 1 − 0.95¹⁰ ≈ 40 %.
- 0.00625.
- 0.01 ≤ 0.05/3 = 0.0167 ✓; 0.02 ≤ 0.05/2 = 0.025 ✓; 0.03 ≤ 0.05/1 ✓. All three. (Bonferroni would keep only the first.)
Try it on your own data: the hypothesis tests tool adjusts a list of p-values with Bonferroni, Holm or Benjamini-Hochberg.
New words: multiple comparisons, family-wise error rate, Bonferroni correction, Holm correction, false discovery rate, Benjamini-Hochberg, pre-registration.
- k tests at α give about 1 − (1 − α)^k chance of at least one false alarm.
- Bonferroni: α/k each. Holm: a step-down version that is never worse.
- FDR methods suit large screens (many metrics); they control the share of false wins.
- Best fix: decide the few segments and metrics before the test.
Check yourself
Sources
- Holm, S. (1979), A Simple Sequentially Rejective Multiple Test Procedure, Scandinavian Journal of Statistics 6(2), 65-70 https://www.jstor.org/stable/4615733
- Benjamini, Y. and Hochberg, Y. (1995), Controlling the False Discovery Rate, Journal of the Royal Statistical Society B 57(1), 289-300 https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.