All lessons

Module 8: Relationships and business data · Lesson 8.3 · about 20 min

Confounding and Simpson's paradox

A version can win in every segment and still lose overall, because the segments are mixed differently. How confounding causes Simpson's paradox, how to spot it, and when to trust the overall number or the split.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 20 minutes · no new maths. You’ll be able to explain Simpson’s paradox with a weighted average, spot when a comparison is confounded, and choose the right level to report.

A trend inside every group can reverse when the groups are added together, because the mix of groups changed; always ask what else differs.

In everyday terms: a school’s average grade can fall even though every class improved, if more pupils joined the class that always scores lower. Nobody got worse; the mix moved.

1. Win every segment, lose overall

Try it: win both, lose overall
phones A / B
computers A / B
overall A / B
On phones, A converts 2.0 % and B 2.4 %; on computers, A 6.0 % and B 6.6 %. B is better on both. But if most of B's visitors are on phones (which convert worse everywhere), B's overall rate can fall below A's. Set both shares equal, as a randomised test would, and the paradox disappears. Made-up numbers.

B beats A on phones (2.4 % vs 2.0 %) and on computers (6.6 % vs 6.0 %). With the starting settings, 80 % of B’s visitors are on phones and only 30 % of A’s, so B’s overall rate (3.24 %) is far below A’s (4.80 %). Set both shares to 50 % and B wins overall too.

This is Simpson’s paradox: a trend in every group reverses (or vanishes) when the groups are combined. It isn’t a trick of arithmetic; it’s what happens when a confounder (here, device) is linked both to which version people got and to the outcome.

2. Where it shows up

  • Observational comparisons: “customers who used the app spend more” (app users are already heavier spenders).
  • Hospital and store league tables: the best hospital looks worst because it takes the hardest cases.
  • Before/after reports: the average order value “fell” because many new, smaller customers joined, while every existing segment spent more.
  • Broken A/B tests: when a mismatch in the mix between groups appears, check for SRM (Lesson 7.6).

3. Which number to trust

There’s no purely statistical answer; it depends on what causes what.

  • If the segment variable is a confounder (it affects both who got the treatment and the outcome, and the treatment didn’t change it), compare within segments and combine with the same weights for both (standardisation).
  • If the segment variable is caused by the treatment (a new page changes which device people finish on), splitting by it can mislead; use the overall comparison.
  • In a clean randomised test the mix is equal by design, so the overall number is fine; segments are for exploring.
Confounder
A third variable linked to both the grouping and the outcome (customer type, store size).
Mix shift
The share of each segment changes between periods or groups.
Weighted average
The overall number is each segment's value weighted by its share, so it moves with the shares.
Physical meaning
An overall average is a blend. Change the recipe (more new customers, who spend less) and the blend changes even if every ingredient got better.

“The overall number tells the true story.” Sometimes it does, sometimes the segments do. Which one answers your question depends on the cause: if the mix shift is the result of your change, the overall number matters; if it is outside noise, compare within segments.

Watch it happen, one step at a time:

The headline. Average order value fell from $62 to $58.
Segments. Returning customers: $70 → $72. New customers: $50 → $48.67. Returning went up.
Mix. Returning were 60 % of orders, now 40 %; new were 40 %, now 60 %.
Recompute. 0.4 × 72 + 0.6 × 48.67 ≈ $58: the drop is the shift towards new customers.
Report. "AOV fell because the customer mix shifted towards new customers, which the campaign intended."
Maths and principle

Principle: an overall rate is a weighted average of segment rates, with weights equal to each group’s mix.

Equation: with share w of a group’s visitors on phones,

poverall=wpphone+(1−w)pcomputer

Check: A: 0.3×2.0%+0.7×6.0%=4.80%. B: 0.8×2.4%+0.2×6.6%=3.24%.

Standardising: use the same weights for both, such as the combined mix (55 % phones): A = 3.80 %, B = 4.29 %. B wins, as it does in each segment.

4. Worked example

Made-up report. “Average order value fell from $62 to $58 this quarter.”

Segment Last quarter This quarter
Returning customers $70 (60 % of orders) $72 (40 % of orders)
New customers $50 (40 % of orders) $48.67 (60 % of orders)
All $62 $58
  • Returning customers spent $2 more; new customers spent about the same ($1.33 less).
  • A marketing campaign brought many new customers, whose orders are smaller, so the mix shifted.
  • At last quarter’s mix, this quarter’s average would be 0.6×72+0.4×48.67=62.67, slightly up.
  • Report: “AOV fell because the mix shifted to new customers; returning customers spent more and new customers about the same.”

5. Practice

  1. Store X has a higher refund rate than store Y overall, but a lower rate for both electronics and clothing. Explain.
  2. In an A/B test, B wins on every device but loses overall. What do you check first?
  3. Should you split a checkout test’s result by “users who used the new express button”?
Answers
  1. X sells far more electronics, which have high refund rates everywhere, so its overall rate is weighted towards the high-refund category.
  2. The split (SRM) and the device mix in each group: in a working test they should match.
  3. No: only B has the button, and who uses it is caused by the treatment. Compare all users in A with all users in B.

Try it on your own data: the cross-tab and chi-square tool builds the segment tables; grouped data gives rates and means per group so you can compare the mix.

New words: Simpson’s paradox, confounder, mix shift, weighted average, standardisation.

  • A total can move opposite to every segment when the mix changes.
  • Overall value = Σ segment share × segment value.
  • Always break key metrics down by the main segments.
  • Decide from the causal story whether the overall or segment view answers your question.

Check yourself

1Version B converts better on phones and on computers, but worse overall. How is that possible?
2In a properly randomised A/B test, should Simpson's paradox appear between segments and the total?
3Which number should guide a decision when the segment split and the total disagree?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.