Module 6: Hypothesis tests · Lesson 6.1 · about 20 min
The logic of a test: null, alternative and evidence
A hypothesis test is a proof by surprise: assume nothing is going on (the null), work out what chance alone would produce, and see whether your result is too unusual to believe. The steps, the words and the courtroom analogy.
Module 6 · about 20 minutes · no new maths. You’ll be able to set up a null and alternative hypothesis, explain how a test weighs evidence, and state a result correctly.
In everyday terms: a court case. The defendant is presumed innocent (the null hypothesis). The evidence has to be too unlikely to fit innocence before the jury convicts. “Not guilty” does not mean “proven innocent”, only “not enough evidence”.
1. Proof by surprise
You run a side-by-side test: 100 visitors choose between the old and new checkout layout, and 62 pick the new one. Is the new layout preferred, or could a 62–38 split happen by chance even if people had no preference?
A hypothesis test answers by assuming the boring explanation and checking whether the data fits it:
- Null hypothesis : nothing is going on. Here, each visitor is equally likely to pick either layout (p = 0.5).
- Alternative : something is going on (p ≠ 0.5).
- Work out what results chance alone would produce under .
- If your result would be rare under , reject . If not, you fail to reject it.
With 5,000 simulated no-preference worlds, a split as lopsided as 62–38 (in either direction) happens about 2 % of the time. Rare, so the “no preference” story looks hard to believe. Try 56: it happens about 27 % of the time, so it’s quite believable under no preference.
2. The courtroom analogy
| Courtroom | Test |
|---|---|
| Presumed innocent | Assume (no effect) |
| Evidence | Data |
| “Beyond reasonable doubt” | Significance level α, often 5 % |
| Guilty | Reject |
| Not guilty (not “innocent”) | Fail to reject |
| Convicting an innocent person | Type I error (Lesson 6.4) |
| Letting a guilty person go | Type II error |
“Not guilty” doesn’t mean the defendant didn’t do it; it means the evidence wasn’t strong enough. Likewise, p = 0.30 doesn’t show the pages are equal; it shows the data can’t tell them apart.
3. The steps every test follows
- State and , and pick α (usually 0.05), before looking at results.
- Choose a test statistic: a number that measures how far the data is from , usually (estimate − null value) ÷ SE.
- Find its distribution under (normal, t, chi-square…).
- Compute the p-value: the chance, under , of a statistic at least as extreme as yours (Lesson 6.2).
- Decide: p ≤ α → reject ; otherwise, don’t. Then report the size of the effect with a confidence interval.
One-sided or two-sided? A two-sided alternative (≠) catches effects in either direction and is the safe default. One-sided (>) is for when only one direction matters and you decided that in advance.
- H₀ (null)
- The boring explanation: no difference, no effect, the claim is true.
- H₁ (alternative)
- What you suspect instead: there is a difference.
- Test statistic
- How far your result is from what H₀ predicts, in standard errors.
- α
- The false-alarm rate you accept in advance, usually 5 %.
- Physical meaning
- A test measures surprise. If the world really were boring, how often would it throw up data this far from boring? Rarely (p below α) means you stop believing the boring story.
“Not significant means there is no effect.” It means the data could not rule out “no effect”. A small or noisy study can miss a real effect. Look at the confidence interval: if it runs from −1 to +5, a sizeable effect is still possible.
Watch it happen, one step at a time:
Maths and principle
Principle: measure how far the result is from the null in units of standard errors, then ask how often chance goes that far.
Equation (most tests):
For the layout test: = 0.62, null 0.5, SE under the null = 0.05, so z = (0.62 − 0.5) ÷ 0.05 = 2.4. Two-sided p = 2 × P(Z > 2.4) ≈ 0.016, close to the simulation (the exact count-based answer is 0.021).
Link to intervals: reject at 5 % exactly when the null value lies outside the 95 % confidence interval.
4. Worked example
Made-up numbers. A supplier claims a mean pack weight of 500 g. You weigh 36 packs: mean 497.6 g, s = 6 g.
- : μ = 500. : μ ≠ 500. α = 0.05.
- SE = 6 ÷ √36 = 1. Statistic t = (497.6 − 500) ÷ 1 = −2.4, with 35 degrees of freedom.
- Two-sided p ≈ 0.022. Below 0.05: reject .
- Effect: packs average 2.4 g light (95 % CI about 0.4 g to 4.4 g light).
Report: “Packs averaged 497.6 g, 2.4 g below the claimed 500 g (95 % CI 0.4 to 4.4 g below, t = −2.4, p = 0.02, n = 36).” Whether 2.4 g matters is a business question, not a statistical one.
5. Practice
- Write and for: “Does the loyalty email change average spend?”
- A test gives p = 0.003 at α = 0.05. What do you conclude, and what should you report alongside it?
- Why must α be chosen before seeing results?
Answers
- : average spend is the same with and without the email. : it differs.
- Reject : the data is very unlikely under no effect. Report the size of the difference and its confidence interval.
- Choosing it after lets you move the goalposts to get the answer you wanted, which makes the error rate meaningless.
Try it on your own data: the hypothesis tests tool runs z, t, proportion and chi-square tests on your numbers and explains each step.
New words: hypothesis test, null hypothesis, alternative hypothesis, test statistic, significance level α, reject / fail to reject, one-sided, two-sided.
- State H₀ and H₁ and choose α before looking at the data.
- Test statistic = (estimate − value under H₀) ÷ SE.
- Small p: data this extreme would be rare if H₀ were true, so reject H₀.
- Failing to reject is not proof of no effect; report the interval.
Check yourself
Sources
- OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 5.3 Hypothesis testing for a proportion https://www.openintro.org/book/os/
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.3 What are statistical tests? https://www.itl.nist.gov/div898/handbook/prc/section1/prc13.htm
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.