All lessons

Module 6: Hypothesis tests · Lesson 6.4 · about 25 min

Type I and type II errors, and power

Every test can go wrong two ways: a false alarm (type I, rate α) or a missed effect (type II, rate β). Power, 1 − β, is the chance of catching a real effect. What sets it, why 80 % is the usual target, and why underpowered tests mislead.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 25 minutes · school maths. You’ll be able to name the two kinds of test error, explain power and what sets it, and see why underpowered studies mislead.

A test can raise a false alarm or miss a real effect; power is the chance it catches an effect of the size you care about.

In everyday terms: a metal detector on a beach. Set it too sensitive and it beeps at bottle caps (type I errors). Set it too dull, or sweep too fast, and you walk past the gold ring (type II errors). Power is how often it beeps when a ring is really there.

1. Two ways to be wrong

Truth: no effect Truth: real effect
Test says “significant” Type I error (false positive), rate α Correct: power = 1 − β
Test says “not significant” Correct Type II error (false negative), rate β

You choose α directly (usually 5 %). β depends on the effect size, the noise and the sample size, so you control it by planning.

Business costs differ: a false positive ships a useless change (wasted build, possible harm); a false negative throws away a good idea. Neither is free.

2. Picturing power

Try it: α, β and power
type I error α (false alarm)
type II error β (missed effect)
power = 1 − β
people per group for 80 % power
Left curve: the difference you would measure if the change did nothing. Right curve: if it truly works. Results past the cut-off (vertical line) are called significant. The second-colour area under the null curve is α, a false alarm. The first-colour area under the real-effect curve, left of the cut-off, is β, a real effect missed. More people pull the curves apart (they get narrower), raising power.

Left curve: where your measured difference lands if there’s no effect. Right curve: where it lands if the effect is real. The cut-off is set by α on the left curve. Power is the share of the right curve beyond the cut-off. Pull the curves apart (bigger effect) or make them narrower (more people, less noise) and power rises.

3. What sets power

  1. Effect size (bigger is easier to detect). You don’t choose the truth, but you choose the smallest effect worth detecting.
  2. Noise (σ): cleaner metrics, paired designs (Lesson 6.3) and adjusting for pre-period behaviour all cut noise.
  3. Sample size n: power rises with n.
  4. α: a looser threshold gives more power but more false alarms.

Convention: plan for 80 % power at α = 5 %, so a real effect of the size you care about is missed only 1 time in 5.

4. Why underpowered tests mislead

With low power:

  • You’ll usually miss real effects, and “not significant” gets misread as “no effect”.
  • When you do get significance, the estimate is usually exaggerated: only lucky, oversized results cross the line (the “winner’s curse”). A test with 20 % power that finds +8 % lift may be looking at a true +3 %.
  • A larger share of your “wins” are false positives (Bayes again, Lesson 2.2: if few ideas truly work and power is low, most significant results are flukes).
Type I error (α)
Calling an effect real when there is none: a false alarm.
Type II error (β)
Missing an effect that is real.
Power = 1 − β
Chance of detecting an effect of a given size.
What raises power
Bigger effect, bigger sample, less noise, larger α.
Physical meaning
An underpowered test is a coin toss dressed as science: even when the change works, it will usually say "no difference", and when it does say "win" the effect looks bigger than it really is.

“We found no significant effect, so the feature does nothing.” With 23 % power, a feature that works exactly as hoped is missed 3 times in 4. Check the power before reading anything into a null result.

Watch it happen, one step at a time:

The goal. Orders average $60 (SD $30). A widget is worth it only for +$3 or more.
Effect in SDs. d = 3 ÷ 30 = 0.1.
The plan. 300 orders per group. SE of the difference = 30 × √(2/300) ≈ 2.45.
Signal over noise. 3 ÷ 2.45 ≈ 1.22 SEs, short of the 1.96 needed.
Power. Φ(1.22 − 1.96) = Φ(−0.74) ≈ 23 %. You need about 1,600 per group for 80 %.
Maths and principle

Principle: power is the probability that the test statistic lands beyond the cut-off when the true effect is δ.

Equation (two groups, means, two-sided, normal approximation): with SE =σ2/n for n per group,

power≈Φ(|δ|SE−z1−α/2)

Solving for 80 % power at α = 0.05 (z values 1.96 and 0.84) gives the planning formula

n per group=2(1.96+0.84)2σ2δ2≈16σ2δ2

Lehr’s rule: about 16/d2 per group, where d=δ/σ. For d = 0.5: 64 per group; d = 0.2: 400; d = 0.1: 1,600.

5. Worked example

Made-up numbers. Average order is $60 with SD $30. A new recommendation widget is worth building only if it lifts the average by at least $3 (5 %).

  • d = 3 ÷ 30 = 0.1. Per group ≈ 16 ÷ 0.01 = 1,600 orders (exact formula: 1,570).
  • The team planned 300 per group. Power then is Φ(3 ÷ (30 × √(2/300)) − 1.96) = Φ(1.22 − 1.96) = Φ(−0.74) ≈ 23 %.
  • So even if the widget works exactly as hoped, a 300-per-group test misses it more than 3 times in 4. Either run longer, cut noise (compare each customer’s order with their own past average), or accept that only a bigger effect would be detectable.

6. Practice

  1. α = 0.05, power = 0.9. What is β?
  2. Using Lehr’s rule, how many per group for d = 0.25?
  3. A test with 25 % power finds a significant +12 % lift. Why be cautious?
Answers
  1. 0.1.
  2. 16 ÷ 0.0625 = 256.
  3. With low power, only unusually large estimates reach significance, so +12 % likely overstates the true effect, and the chance it’s a false positive is higher than with a well-powered test.

Try it on your own data: the sample size calculator plans n for a chosen power, and the hypothesis tests tool reports power for a finished test.

New words: type I error, type II error, false positive, false negative, power, β, minimum detectable effect, winner’s curse.

  • α = false-alarm rate; β = miss rate; power = 1 − β.
  • Power grows with effect size, sample size and lower noise.
  • Plan for 80 % power at the smallest effect worth acting on.
  • Low power makes both misses and exaggerated “wins” more likely.

Check yourself

1A test declares a new page better when it truly makes no difference. Which error is that?
2A test has 30 % power for the effect you care about. What does that mean?
3Which change raises power without raising the false alarm rate?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.