All lessons

Module 8: Relationships and business data · Lesson 8.1 · about 20 min

Correlation: Pearson and Spearman

Measure how two numbers move together with Pearson's r and Spearman's rank correlation, test whether r could be zero, put an interval on it, and avoid the traps: outliers, curves, huge samples and hidden causes.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 20 minutes · school maths. You’ll be able to compute and test a correlation, choose Pearson or Spearman, and explain why correlation alone doesn’t show cause.

Correlation measures how closely two numbers move together on a straight line, from −1 to +1, and says nothing on its own about cause.

In everyday terms: ice-cream sales and sunburn rise and fall together, but ice cream does not cause sunburn. Summer causes both. Correlation spots the dance; it does not say who leads.

1. Pearson’s r

Pearson’s r measures how closely two numbers follow a straight line: +1 a perfect rising line, −1 a perfect falling line, 0 no straight-line relationship.

Try it: set the correlation
measured r
r² (share of spread shared)
r runs from −1 (a perfect downhill line) through 0 (no straight-line link) to +1 (a perfect uphill line). It measures only straight-line patterns, and a single far-away point can change it a lot. Always look at the scatter plot, not just the number.

Add the far-away point and watch one dot move r a long way. That’s Pearson’s weakness.

r² is the share of one variable’s spread “shared” with the other: r = 0.6 means r² = 36 %.

2. Spearman’s rank correlation

Replace each value by its rank (1st, 2nd, …) and compute Pearson’s r on the ranks. Spearman’s ρ measures any steadily rising or falling relationship, straight or curved, and one extreme value can only be “the largest”, so it can’t dominate. Use it for skewed data (revenue, house prices), ranks and ratings, or when outliers are present.

3. Is r real? Test and interval

The test of “true correlation is zero” uses a t-statistic with n−2 degrees of freedom. For an interval, use Fisher’s z transform.

Two warnings:

  • Small samples give wild r values. With 10 points, r = 0.6 still has a 95 % CI from about −0.05 to 0.89.
  • Huge samples make tiny correlations “significant”. r = 0.1 from 2,000 customers has p < 0.0001 but explains 1 % of the spread. Report r and its interval, not just the p-value.

4. Correlation is not causation

Try it: reveal the hidden cause
correlation r of the dots shown
days shown
Over a summer, days with more ice cream sold also have more swimming accidents (r is high). Ice cream does not cause drowning: hot weather drives both. Colour by temperature and the pattern lines up with heat; keep only days of similar temperature and the link almost disappears. A hidden cause like this is called a confounder. Made-up data.

A confounder drives both variables. Here hot days raise both ice cream sales and swimming accidents. Show only days in one temperature band and the correlation mostly vanishes. Other explanations for a correlation: reverse cause (busier stores hire more staff, not the other way round), selection (only successful products survive into your data), and chance across many comparisons. Experiments (Module 7) are the clean way to show cause.

Pearson r
Average product of z-scores: how well a straight line fits, from −1 to +1.
r²
Share of the variation in one variable lined up with the other.
Spearman ρ
Pearson on the ranks: any steady up-or-down trend, robust to outliers.
Physical meaning
r compares each point's position with the averages. Points in the "both above" and "both below" corners push r up; points in the mixed corners push it down. One extreme point sitting far out in a corner can create or hide a correlation on its own.

“r = 0 means the variables are unrelated.” r only measures straight-line patterns. A U-shape (sales highest at medium prices) can have r near 0 and a very strong relationship. Always plot first.

Watch it happen, one step at a time:

The data. 50 stores: staff hours per week and satisfaction score.
Plot. A loose upward cloud, plus one big flagship store top right.
Pearson. r = 0.30, p = 0.034, CI 0.02 to 0.53.
Check the flagship. Without it, r = 0.22. Spearman (all stores): 0.24.
Report. A weak positive link, not proof that more hours raise scores: bigger, busier stores differ in other ways.
Maths and principle

Principle: r is the average product of the two variables in z-score units.

Equation:

r=∑(xi−x‾)(yi−y‾)∑(xi−x‾)2∑(yi−y‾)2,t=rn−21−r2

Interval: z=tanh−1⁡(r) is roughly normal with SE 1/n−3; build z±1.96SE and transform back with tanh⁡.

Check: r = 0.3, n = 50: t = 2.18, p = 0.034; 95 % CI 0.02 to 0.53 (scipy agrees; tests/stats.mjs checks the t and p).

5. Worked example

Made-up data: 50 stores; correlation between staff hours per week and customer satisfaction score.

  • Scatter plot first: a loose upward cloud, one very large flagship store at the top right.
  • Pearson r = 0.30, p = 0.034, 95 % CI 0.02 to 0.53.
  • Without the flagship: r drops to 0.22. Spearman (all stores): 0.24.
  • Report: “a weak positive association (Spearman 0.24); stores with more staff hours tend to score a little higher. This doesn’t show that adding hours would raise scores: bigger, busier stores may differ in other ways.”

6. Practice

  1. r = −0.8. Describe it in words.
  2. Which correlation would you use for customers’ spend (very skewed) against visits?
  3. r = 0.05, n = 100,000, p < 0.001. Is it important?
Answers
  1. A strong negative straight-line relationship: as one goes up, the other tends to go down, and r² = 64 % of the spread is shared.
  2. Spearman, because spend is skewed and a few big spenders would dominate Pearson’s r.
  3. It’s very likely real but tiny: r² = 0.25 %. Statistically significant, practically negligible.

Try it on your own data: the CSV profiler tool shows correlations between the numeric columns of a file; regression workbench fits the line and gives r with a scatter plot; bootstrap test puts an interval on any statistic, including Spearman’s ρ.

New words: Pearson’s r, Spearman’s rank correlation, r², Fisher’s z, confounder, reverse causation.

  • Plot first; r measures straight-line association only.
  • r ranges from −1 to +1; r² is the share of variation lined up.
  • Spearman uses ranks and resists outliers.
  • Correlation is not causation: look for confounders and use experiments when you can.

Check yourself

1r = 0.3 from 50 stores. Is that evidence of a real (non-zero) correlation at α = 0.05?
2One huge store sits far from the rest and drives r from 0.1 to 0.7. What should you report?
3Ice cream sales and drownings are correlated. Why?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.