Module 5: Decision trees · Lesson 5.2 · about 20 min
Purity: Gini impurity and entropy
To choose its questions a tree must measure how mixed a group is. Gini impurity and entropy both score a pure group 0 and a 50/50 mix highest. Work them out by hand and mix a box to see the two curves.
Module 5 · about 20 minutes · read first: 1.6, 5.1. You’ll be able to compute Gini impurity and entropy for any group by hand and explain what they measure.
1. Why this matters
Every split in a decision tree, random forest and gradient-boosting model is chosen with one of these two numbers. They appear in the settings of every tree library (criterion="gini" or "entropy"). Entropy is also the quantity that measures information in a message and, as cross-entropy (Lesson 4.3), the loss used to train language models.
2. What “pure” means
A box (the group of training rows at a node) is pure if every row has the same class. The tree wants boxes as pure as possible, because then its leaf answers are confident and right. So it needs one number for “how mixed”: 0 for pure, largest for an even mix.
With two classes, let be the share of class 1 in the box.
- Gini impurity: . It is the chance that two rows picked at random from the box (putting the first back before picking the second) have different classes.
- Entropy: , measured in bits. It is about how many yes/no questions, on average, you’d need to learn the class of a random row. ( asks “2 to what power gives this number?”: .)
Both are 0 at and , and peak at : 0.5 for Gini, 1 bit for entropy. Their curves have almost the same shape, so they almost always pick the same splits.
A box of 10 emails: 6 spam, 4 normal. The share of spam is p = 0.6.
Square the shares. 0.6² = 0.36 and 0.4² = 0.16. Together 0.52: the chance two random picks agree.
Gini. 1 − 0.52 = 0.48: the chance two random picks disagree. Close to the 0.5 maximum, so this box is quite mixed.
Entropy. Spam's surprise is −log₂ 0.6 = 0.737 bits, normal's −log₂ 0.4 = 1.322 bits. Average them by share: 0.6 × 0.737 + 0.4 × 1.322 = 0.971 bits.
After a good question the box splits into 5 all-spam (Gini 0) and 5 with 1 spam and 4 normal (Gini 0.32). Weighted Gini: 0.16. Much tidier.
- pk
- The share of rows in the box that belong to class k.
- Gini = 1 − Σ pk²
- The chance that two rows drawn at random (with replacement) are of different classes.
- H = −Σ pk log₂ pk
- Entropy: the average surprise, in bits, when you draw one row and look at its class.
- Physical meaning
- A bag of sweets. A bag of only red ones holds no surprise when you draw one; a half-red, half-green bag holds the most. Gini and entropy are two rulers for that surprise.
3. More than two classes
With classes at shares , add up over all of them:
The maximum grows with : an even mix of 4 classes has Gini 0.75 and entropy 2 bits.
“Entropy is better than Gini because it comes from information theory.” In practice they nearly always choose the same split, because their curves have almost the same shape. Gini is a little faster to compute (no logarithms), which is why many libraries use it by default. Pick either and spend your effort on depth and leaf size instead.
Maths and principle
Principle: impurity measures how uncertain we are about the class of a random row in the box. A good split lowers it.
Why Gini is “chance of disagreeing”: pick two rows independently. Both are class with probability , so they agree with probability and disagree with probability .
Why entropy is “bits”: a class with share is as surprising as coin flips (share 1/8 = 3 bits). Entropy is the average surprise:
By convention , so a class that is absent adds nothing.
Try it: a box has 8 squares and 2 circles. Gini? Entropy? (. Gini = 1 − 0.64 − 0.04 = 0.32. Entropy = −0.8 log₂ 0.8 − 0.2 log₂ 0.2 = 0.258 + 0.464 = 0.722 bits.)
4. Worked example
Made-up numbers. A node holds 10 emails, 6 spam and 4 normal, so .
- Gini = 1 − 0.6² − 0.4² = 1 − 0.36 − 0.16 = 0.48.
- Entropy = −0.6 log₂ 0.6 − 0.4 log₂ 0.4 = 0.6 × 0.737 + 0.4 × 1.322 = 0.442 + 0.529 = 0.971 bits.
Now a question splits them into a left box of 5 (all spam) and a right box of 5 (1 spam, 4 normal):
- Left: pure, Gini 0, entropy 0.
- Right: , Gini = 1 − 0.04 − 0.64 = 0.32, entropy = 0.722.
Each box holds half the rows, so the weighted impurity after the split is 0.5 × 0 + 0.5 × 0.32 = 0.16 (Gini), or 0.5 × 0 + 0.5 × 0.722 = 0.361 (entropy). The split cut Gini from 0.48 to 0.16. Lesson 5.3 turns that drop into the rule for choosing splits.
5. Practice
- Gini impurity of a box with 3 squares and 1 circle?
- Which is more mixed: a 70/30 box or a 40/60 box?
- A box has three classes in equal shares. Find its Gini impurity.
Answers
- : 1 − 0.5625 − 0.0625 = 0.375.
- 40/60, because it is closer to 50/50 (Gini 0.48 against 0.42).
- 1 − 3 × (1/3)² = 1 − 1/3 ≈ 0.667.
- A pure box (one class only) has Gini 0 and entropy 0.
- Gini = 1 − Σ pk²: the chance two random rows disagree. Two classes peak at 0.5.
- Entropy = −Σ pk log₂ pk: the average surprise in bits. Two classes peak at 1 bit.
- The two curves have nearly the same shape, so trees built with either are almost the same.
Try it on your own data: the small model trainer grows a tree on your CSV with Gini impurity or entropy; switch between them and see how little the tree changes.
New words: purity, Gini impurity, entropy, bit, surprise.
Check yourself
Sources
- Shannon (1948), A mathematical theory of communication, Bell System Technical Journal 27(3), 379-423 https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
- James, Witten, Hastie and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition, Springer, Section 8.1.2 (Gini index and entropy) https://www.statlearning.com/
- scikit-learn user guide: Decision trees, mathematical formulation (classification criteria) https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.