Module 2: Looking at data before modelling · Lesson 2.4 · about 25 min
Cleaning: missing values, outliers and scaling
Real data has gaps, errors and columns on wildly different scales. Learn the safe ways to fill gaps, when to keep or fix outliers, and why distance-based models need scaling, with two simulations.
Module 2 · about 25 minutes · read first: 1.6, 2.2. You’ll be able to choose a way to handle missing values, decide what to do with an outlier, and standardise or min-max scale a column.
1. Why this matters
Data scientists often say they spend most of their time cleaning. It is like washing and chopping vegetables before cooking: dull, but skip it and the meal is ruined. Models can’t use blank cells, are thrown by impossible values, and many quietly favour whichever column has the biggest numbers. A few careful steps here fix more problems than any clever model later.
2. Missing values
First ask why a value is missing. If a sensor was broken at random, the gap carries no information. If customers skip the income question when their income is very high or very low, the gap itself is a clue. Then choose:
| Method | When it is reasonable | The cost |
|---|---|---|
| Drop the rows | Few rows affected, missing at random | You lose those rows’ other columns too |
| Drop the column | Mostly empty, or not useful | You lose the feature |
| Fill with mean or median | A modest share is missing | Shrinks the spread; hides uncertainty |
| Fill with the most common category | Categorical columns | Over-counts that category |
| Fill from similar rows (k-nearest neighbours) or a model | Gaps follow a pattern | More work; can leak (see below) |
| Add a “was missing” flag | Always worth it alongside a fill | One extra column |
Important: work out fill values (means, medians) from the training rows only, then apply them to the test rows. Using the whole dataset lets test information slip into training: a small case of leakage (Lesson 7.4).
3. Outliers
An outlier is a value far from the rest. Tukey’s rule from Lesson 2.2 flags values beyond 1.5 IQR outside the box. Then decide which kind it is:
- An error: a height of 17 m, a negative age, hours typed as minutes. Fix it if you can tell what was meant, otherwise set it to missing.
- Real but extreme: a $40 million house, a customer with 900 orders. Keep it. If it overwhelms a squared-error model, consider capping (also called winsorising: set everything above the 99th percentile to the 99th percentile), using a log scale, or a model that copes (trees, Module 5).
- The point of the study: fraud, machine faults, rare diseases. Never delete these. Finding them is the job (Lesson 8.5).
“Outliers are bad data, so delete them.” Some are errors, and those should be fixed. But many are real, and some are exactly what you are looking for. A fraud model trained on data with the strangest payments deleted has been taught that fraud doesn’t exist. Check each outlier, fix the impossible ones, and keep the real ones.
4. Scaling
Columns come in different units and sizes. Models that use distances (k-nearest neighbours, k-means, support vector machines) or gradient descent (linear and logistic regression, neural networks) are thrown off by that. Pick a customer below and compare raw and scaled distances.
Tap a customer to choose them.
Two common scalings:
- Standardisation (z-scores): subtract the mean, divide by the standard deviation. Each column then has mean 0 and standard deviation 1.
- Min-max scaling: shift and stretch to the range 0 to 1.
Here is standardisation step by step, for two customers described by age and income:
Trees (Module 5) and forests don’t need scaling, because they only ask “is x above or below this value?”, and scaling doesn’t change the order of values.
Maths and principle
Principle: scaling changes the units of each column so that no column dominates just because its numbers are big.
Equations:
Why distance needs it: the straight-line distance between two rows is
With a 30-year age gap and a $5,000 income gap, . Age adds less than 0.1 to the distance. After standardising (say years and ), the gaps become 2.1 and 0.23, and age now counts most.
Analogy: comparing two runners by height in metres and weight in grams. Weight wins every argument purely because of the unit.
Try it: a column has min 10 and max 50. Min-max scale 20. (.)
- x̄ and s
- The mean and standard deviation of the column, worked out on the training rows.
- z = (x − x̄) ÷ s
- How many standard deviations a value sits above (positive) or below (negative) the average.
- x′ = (x − min) ÷ (max − min)
- Where a value sits between the smallest (0) and largest (1) in the training data.
- Imputation
- Filling a blank with a sensible stand-in, such as the training median.
- Physical meaning
- Scaling swaps each column's own units (years, dollars, grams) for one shared unit: "how unusual is this?". A z of 2 means "quite unusual" whether it is a height, a price or a temperature, so no column wins an argument just because its unit is small.
5. Worked example
Made-up numbers. A clinic table of 2,000 patients has: age (40 blanks), weight in kg (3 values of 0, 1 of 700), blood pressure, and smoker (yes/no, 120 blanks).
- Weight 0 and 700: errors. Set them to missing.
- Age: 2 % missing, no pattern found. Fill with the training-set median (52) and add
age_missing. - Smoker: 6 % missing, and the blanks are mostly from one clinic that didn’t ask. Add a third category “unknown” rather than guessing “no”.
- Scaling: standardise age, weight and blood pressure using training means and standard deviations, because the next step uses logistic regression.
6. Practice
- Standardise 130 when the mean is 100 and the standard deviation is 20.
- Should you delete the 50 most unusual credit card payments before training a fraud model?
- Why compute the fill value from training rows only?
Answers
- z = (130 − 100) / 20 = 1.5.
- No. Unusual payments are exactly what the model must learn about. Check them for errors, but keep the real ones.
- Because the test set must stand in for future data the model has never seen. Using it to compute fills lets its information leak into training.
Try it on your own data: the CSV profiler counts blanks per column and flags outliers.
New words: missing at random, imputation (filling gaps), missing flag, capping or winsorising, standardisation, z-score, min-max scaling, leakage.
- Ask why a value is missing; fill with the median or most common value and add a “was missing” flag.
- Fix outliers that are errors; keep real extremes, and never delete the cases you are trying to find.
- Standardise (z = (x − mean) ÷ sd) or min-max scale for models that use distances or gradient descent; trees don’t need it.
- Work out every fill value, mean and sd from the training rows only, to avoid leakage.
Check yourself
Sources
- Little, R. J. A. and Rubin, D. B. (2019), Statistical Analysis with Missing Data, 3rd ed., Wiley, Chapter 1 (why data are missing)
- scikit-learn user guide: Imputation of missing values https://scikit-learn.org/stable/modules/impute.html
- scikit-learn user guide: Preprocessing data (standardisation and min-max scaling) https://scikit-learn.org/stable/modules/preprocessing.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.