Module 6: Hypothesis tests · Lesson 6.7 · about 20 min
When the data is not normal: rank tests
Rank tests replace values with their positions in order, so outliers and skew lose their grip: Mann-Whitney U for two groups, Wilcoxon signed-rank for paired data, Kruskal-Wallis for several groups. When to use them, and what they actually test.
Module 6 · about 20 minutes · no new maths. You’ll be able to choose a rank test, explain how it handles outliers, and say what it tests.
In everyday terms: judging a race by finishing order instead of times. If one runner stops for a picnic, their time is ridiculous but their place is simply “last”. The ranking still says who was generally faster.
1. Why ranks
t-tests compare means, and means are pulled about by outliers and long tails. With small samples of skewed data (session times, spend, waiting times), one extreme value can decide the result. Rank tests (non-parametric tests) replace each value by its position in the sorted data, so the biggest value is just “the biggest”, however big.
Drag B’s top value to 300. The mean difference balloons, yet the t-test’s p-value gets worse, because that one value inflates B’s standard deviation even more. The Mann-Whitney test doesn’t move: B’s values are still mostly above A’s.
2. The three tests
| Situation | t-based test | Rank test |
|---|---|---|
| Two independent groups | Welch t | Mann-Whitney U (Wilcoxon rank-sum) |
| Paired / before-after | Paired t | Wilcoxon signed-rank |
| Three or more groups | One-way ANOVA | Kruskal-Wallis |
Mann-Whitney U, in words: pool both groups and rank everything. If the groups are alike, their average ranks should be similar. U counts, over all pairs (one from each group), how often a value from one group beats one from the other. U / (n₁n₂) is directly “the chance a random B beats a random A”, a handy effect size.
3. What they test (careful)
Rank tests don’t test means. Mann-Whitney tests whether values in one group tend to be larger than in the other (stochastic dominance). If the two groups have the same shape and spread, that’s the same as a difference in medians. Report medians and the “probability of superiority”, not means.
4. When to use which
- Small samples (under ~30 per group) of skewed data or with outliers: rank test.
- Ordinal data (ratings 1 to 5, satisfaction levels): rank test; means of ratings are debatable.
- Large samples and you care about the mean (revenue per visitor, where totals matter): t-test or bootstrap. The mean is what drives revenue, and the CLT protects you.
- Rank tests lose only a little power when the data really is normal (about 5 %), so they’re a safe choice when unsure.
- Rank
- Position after sorting all values together: 1 for the smallest.
- Mann-Whitney U
- For two groups: counts how often a value from one group beats a value from the other.
- Wilcoxon signed-rank
- For paired data: ranks the sizes of the differences.
- Kruskal-Wallis
- For three or more groups: the rank version of ANOVA.
- Physical meaning
- U ÷ (n₁ × n₂) is the probability that a random member of one group beats a random member of the other: a plain-language effect size managers understand.
“Rank tests compare medians.” Strictly they compare whole distributions: whether one group tends to have larger values. They line up with medians only when the two groups have the same shape.
Watch it happen, one step at a time:
Maths and principle
Principle: ranks keep the order of the data and throw away the distances, so no single value can dominate.
Mann-Whitney: with rank sum for group 1, and . For larger samples, under the null is roughly normal with mean and SD (with a correction for ties).
In code: Python scipy.stats.mannwhitneyu(b, a), wilcoxon(after, before), kruskal(a, b, c). Excel has no built-in; rank with RANK.AVG and use the formulas above.
5. Worked example
Made-up numbers. Minutes on a help page: A = 3, 5, 9; B = 6, 10, 12, 40.
- Pool and rank: 3(1), 5(2), 6(3), 9(4), 10(5), 12(6), 40(7).
- A’s ranks: 1, 2, 4. R_A = 7. U_A = 7 − 3 × 4 ÷ 2 = 1. U_B = 3 × 4 − 1 = 11.
- Of the 12 A–B pairs, B wins 11: the probability a random B beats a random A is 11/12 ≈ 92 %.
- With samples this small, use exact tables: two-sided p ≈ 0.11. Suggestive, not conclusive.
- The 40 counts only as “the largest”. With a t-test, it would inflate B’s SD so much that p would be about 0.24.
6. Practice
- Rank the values 7, 2, 9, 2, 5 (ties share the average rank).
- Which test for: customer ratings 1–5 for two app versions, 25 people each?
- A test of revenue per visitor with 50,000 per group: rank test or t-test?
Answers
- 2 and 2 share ranks 1 and 2 → 1.5 each; 5 → 3; 7 → 4; 9 → 5. So 4, 1.5, 5, 1.5, 3.
- Mann-Whitney (ordinal ratings, modest samples).
- Usually a t-test (or bootstrap) on the mean: revenue totals depend on the mean, and with 50,000 per group the CLT makes it reliable.
Try it on your own data: the hypothesis tests tool includes Mann-Whitney, Wilcoxon and Kruskal-Wallis, and the bootstrap test tool offers permutation tests that also avoid the normal assumption.
New words: non-parametric test, rank, Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis, probability of superiority, ordinal data.
- Rank tests work on order, so outliers and skew lose their grip.
- Two groups: Mann-Whitney U; paired: Wilcoxon signed-rank; 3+ groups: Kruskal-Wallis.
- U ÷ (n₁n₂) = probability a random value from one group beats the other.
- Use them for small or skewed samples; with large normal-ish data t-tests are slightly more powerful.
Check yourself
Sources
- Hollander, M., Wolfe, D. A. and Chicken, E. (2014), Nonparametric Statistical Methods, 3rd edition, Wiley
- Mann, H. B. and Whitney, D. R. (1947), On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other, Annals of Mathematical Statistics 18(1), 50-60 https://doi.org/10.1214/aoms/1177730491
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.