All lessons

Module 6: Hypothesis tests · Lesson 6.1 · about 20 min

The logic of a test: null, alternative and evidence

A hypothesis test is a proof by surprise: assume nothing is going on (the null), work out what chance alone would produce, and see whether your result is too unusual to believe. The steps, the words and the courtroom analogy.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 20 minutes · no new maths. You’ll be able to set up a null and alternative hypothesis, explain how a test weighs evidence, and state a result correctly.

A test assumes nothing is going on, works out how often chance alone would give a result like yours, and calls the result real only if that is rare.

In everyday terms: a court case. The defendant is presumed innocent (the null hypothesis). The evidence has to be too unlikely to fit innocence before the jury convicts. “Not guilty” does not mean “proven innocent”, only “not enough evidence”.

1. Proof by surprise

You run a side-by-side test: 100 visitors choose between the old and new checkout layout, and 62 pick the new one. Is the new layout preferred, or could a 62–38 split happen by chance even if people had no preference?

A hypothesis test answers by assuming the boring explanation and checking whether the data fits it:

  1. Null hypothesis H0: nothing is going on. Here, each visitor is equally likely to pick either layout (p = 0.5).
  2. Alternative H1: something is going on (p ≠ 0.5).
  3. Work out what results chance alone would produce under H0.
  4. If your result would be rare under H0, reject H0. If not, you fail to reject it.
Try it: what would chance alone do?
worlds simulated
as extreme as yours (either way)
so the p-value is about
Each simulated world flips a fair coin for each of 100 visitors, because under the null hypothesis nobody prefers either layout. The bars show what chance alone produces. Results at least as far from 50 as yours (in either direction) are highlighted; their share is the p-value. If chance rarely gets that far, the "no preference" story looks hard to believe.

With 5,000 simulated no-preference worlds, a split as lopsided as 62–38 (in either direction) happens about 2 % of the time. Rare, so the “no preference” story looks hard to believe. Try 56: it happens about 27 % of the time, so it’s quite believable under no preference.

2. The courtroom analogy

Courtroom Test
Presumed innocent Assume H0 (no effect)
Evidence Data
“Beyond reasonable doubt” Significance level α, often 5 %
Guilty Reject H0
Not guilty (not “innocent”) Fail to reject H0
Convicting an innocent person Type I error (Lesson 6.4)
Letting a guilty person go Type II error

“Not guilty” doesn’t mean the defendant didn’t do it; it means the evidence wasn’t strong enough. Likewise, p = 0.30 doesn’t show the pages are equal; it shows the data can’t tell them apart.

3. The steps every test follows

  1. State H0 and H1, and pick α (usually 0.05), before looking at results.
  2. Choose a test statistic: a number that measures how far the data is from H0, usually (estimate − null value) ÷ SE.
  3. Find its distribution under H0 (normal, t, chi-square…).
  4. Compute the p-value: the chance, under H0, of a statistic at least as extreme as yours (Lesson 6.2).
  5. Decide: p ≤ α → reject H0; otherwise, don’t. Then report the size of the effect with a confidence interval.

One-sided or two-sided? A two-sided alternative (≠) catches effects in either direction and is the safe default. One-sided (>) is for when only one direction matters and you decided that in advance.

H₀ (null)
The boring explanation: no difference, no effect, the claim is true.
H₁ (alternative)
What you suspect instead: there is a difference.
Test statistic
How far your result is from what H₀ predicts, in standard errors.
α
The false-alarm rate you accept in advance, usually 5 %.
Physical meaning
A test measures surprise. If the world really were boring, how often would it throw up data this far from boring? Rarely (p below α) means you stop believing the boring story.

“Not significant means there is no effect.” It means the data could not rule out “no effect”. A small or noisy study can miss a real effect. Look at the confidence interval: if it runs from −1 to +5, a sizeable effect is still possible.

Watch it happen, one step at a time:

The claim. A supplier says packs weigh 500 g on average. You weigh 36: mean 497.6 g, SD 6 g.
Assume the claim (H₀). If μ = 500, sample means of 36 packs wobble with SE 6 ÷ √36 = 1 g.
Measure the surprise. 497.6 is 2.4 g below 500: t = −2.4 SEs.
How rare? A t at least 2.4 away from 0 in either direction happens about 2.2 % of the time by chance.
Decide. 2.2 % is below 5 %: reject the claim. Packs average 2.4 g light (95 % CI 0.4 to 4.4 g).
Maths and principle

Principle: measure how far the result is from the null in units of standard errors, then ask how often chance goes that far.

Equation (most tests):

test statistic=estimate−value under H0standard error

For the layout test: pˆ = 0.62, null 0.5, SE under the null 0.5×0.5/100 = 0.05, so z = (0.62 − 0.5) ÷ 0.05 = 2.4. Two-sided p = 2 × P(Z > 2.4) ≈ 0.016, close to the simulation (the exact count-based answer is 0.021).

Link to intervals: reject H0 at 5 % exactly when the null value lies outside the 95 % confidence interval.

4. Worked example

Made-up numbers. A supplier claims a mean pack weight of 500 g. You weigh 36 packs: mean 497.6 g, s = 6 g.

  • H0: μ = 500. H1: μ ≠ 500. α = 0.05.
  • SE = 6 ÷ √36 = 1. Statistic t = (497.6 − 500) ÷ 1 = −2.4, with 35 degrees of freedom.
  • Two-sided p ≈ 0.022. Below 0.05: reject H0.
  • Effect: packs average 2.4 g light (95 % CI about 0.4 g to 4.4 g light).

Report: “Packs averaged 497.6 g, 2.4 g below the claimed 500 g (95 % CI 0.4 to 4.4 g below, t = −2.4, p = 0.02, n = 36).” Whether 2.4 g matters is a business question, not a statistical one.

5. Practice

  1. Write H0 and H1 for: “Does the loyalty email change average spend?”
  2. A test gives p = 0.003 at α = 0.05. What do you conclude, and what should you report alongside it?
  3. Why must α be chosen before seeing results?
Answers
  1. H0: average spend is the same with and without the email. H1: it differs.
  2. Reject H0: the data is very unlikely under no effect. Report the size of the difference and its confidence interval.
  3. Choosing it after lets you move the goalposts to get the answer you wanted, which makes the error rate meaningless.

Try it on your own data: the hypothesis tests tool runs z, t, proportion and chi-square tests on your numbers and explains each step.

New words: hypothesis test, null hypothesis, alternative hypothesis, test statistic, significance level α, reject / fail to reject, one-sided, two-sided.

  • State H₀ and H₁ and choose α before looking at the data.
  • Test statistic = (estimate − value under H₀) ÷ SE.
  • Small p: data this extreme would be rare if H₀ were true, so reject H₀.
  • Failing to reject is not proof of no effect; report the interval.

Check yourself

1A new page is tested against the old one. What is the null hypothesis?
2A test gives p = 0.30. What should you conclude?
3Why is a two-sided test the usual default?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.