Home / Learn / Data & AI / ML foundations / Lesson 2.4
All lessons

Module 2: Looking at data before modelling · Lesson 2.4 · about 25 min

Cleaning: missing values, outliers and scaling

Real data has gaps, errors and columns on wildly different scales. Learn the safe ways to fill gaps, when to keep or fix outliers, and why distance-based models need scaling, with two simulations.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 2 · about 25 minutes · read first: 1.6, 2.2. You’ll be able to choose a way to handle missing values, decide what to do with an outlier, and standardise or min-max scale a column.

Fill gaps carefully, fix errors but keep real extremes, and put columns on the same scale, always using numbers worked out from the training rows only.

1. Why this matters

Data scientists often say they spend most of their time cleaning. It is like washing and chopping vegetables before cooking: dull, but skip it and the meal is ruined. Models can’t use blank cells, are thrown by impossible values, and many quietly favour whichever column has the biggest numbers. A few careful steps here fix more problems than any clever model later.

2. Missing values

First ask why a value is missing. If a sensor was broken at random, the gap carries no information. If customers skip the income question when their income is very high or very low, the gap itself is a clue. Then choose:

Method When it is reasonable The cost
Drop the rows Few rows affected, missing at random You lose those rows’ other columns too
Drop the column Mostly empty, or not useful You lose the feature
Fill with mean or median A modest share is missing Shrinks the spread; hides uncertainty
Fill with the most common category Categorical columns Over-counts that category
Fill from similar rows (k-nearest neighbours) or a model Gaps follow a pattern More work; can leak (see below)
Add a “was missing” flag Always worth it alongside a fill One extra column
Try it: fill the gaps
rows left
mean age
standard deviation

Hollow rings are filled-in values. Dropping rows throws away the other columns of those rows too. Filling with the mean or median keeps the rows but squeezes the spread, because every gap gets the same middle value. Filling with 0 invents babies and drags the mean down. Whatever you choose, add a yes/no column "age was missing", because missing is often a clue in itself. Made-up ages.

Important: work out fill values (means, medians) from the training rows only, then apply them to the test rows. Using the whole dataset lets test information slip into training: a small case of leakage (Lesson 7.4).

3. Outliers

An outlier is a value far from the rest. Tukey’s rule from Lesson 2.2 flags values beyond 1.5 IQR outside the box. Then decide which kind it is:

  • An error: a height of 17 m, a negative age, hours typed as minutes. Fix it if you can tell what was meant, otherwise set it to missing.
  • Real but extreme: a $40 million house, a customer with 900 orders. Keep it. If it overwhelms a squared-error model, consider capping (also called winsorising: set everything above the 99th percentile to the 99th percentile), using a log scale, or a model that copes (trees, Module 5).
  • The point of the study: fraud, machine faults, rare diseases. Never delete these. Finding them is the job (Lesson 8.5).

“Outliers are bad data, so delete them.” Some are errors, and those should be fixed. But many are real, and some are exactly what you are looking for. A fraud model trained on data with the strangest payments deleted has been taught that fraud doesn’t exist. Check each outlier, fix the impossible ones, and keep the real ones.

4. Scaling

Columns come in different units and sizes. Models that use distances (k-nearest neighbours, k-means, support vector machines) or gradient descent (linear and logistic regression, neural networks) are thrown off by that. Pick a customer below and compare raw and scaled distances.

Try it: who are the nearest neighbours?

Tap a customer to choose them.

Distance adds up the differences in every column. On raw numbers a gap of $5,000 counts as 5,000 while a gap of 30 years counts as 30, so age is ignored and "nearest" just means "similar income" (the lines run sideways). After scaling, both columns have a similar range and both count. Models that use distance or gradient descent need scaled inputs. Made-up customers.

Two common scalings:

  • Standardisation (z-scores): subtract the mean, divide by the standard deviation. Each column then has mean 0 and standard deviation 1.
  • Min-max scaling: shift and stretch to the range 0 to 1.

Here is standardisation step by step, for two customers described by age and income:

Raw gaps. Customer A is 25 with income $50,000; B is 55 with $55,000. Gaps: 30 years and $5,000.
Raw distance. √(30² + 5000²) ≈ 5000.09. Age adds less than 0.1: the model would barely notice a 30-year age gap.
Divide by the spreads. In the training data, age has standard deviation 14 years and income $22,000. So the gaps become 30 ÷ 14 = 2.14 and 5000 ÷ 22000 = 0.23 standard deviations.
Scaled distance. √(2.14² + 0.23²) ≈ 2.15. Now the age gap, which really is large, counts most.

Trees (Module 5) and forests don’t need scaling, because they only ask “is x above or below this value?”, and scaling doesn’t change the order of values.

Maths and principle

Principle: scaling changes the units of each column so that no column dominates just because its numbers are big.

Equations:

z=x−x‾sx′=x−xmin⁡xmax⁡−xmin⁡

Why distance needs it: the straight-line distance between two rows is

d=(Δage)2+(Δincome)2

With a 30-year age gap and a $5,000 income gap, d=302+50002≈5000.09. Age adds less than 0.1 to the distance. After standardising (say sage=14 years and sincome=$22,000), the gaps become 2.1 and 0.23, and age now counts most.

Analogy: comparing two runners by height in metres and weight in grams. Weight wins every argument purely because of the unit.

Try it: a column has min 10 and max 50. Min-max scale 20. ((20−10)/40=0.25.)

x̄ and s
The mean and standard deviation of the column, worked out on the training rows.
z = (x − x̄) ÷ s
How many standard deviations a value sits above (positive) or below (negative) the average.
x′ = (x − min) ÷ (max − min)
Where a value sits between the smallest (0) and largest (1) in the training data.
Imputation
Filling a blank with a sensible stand-in, such as the training median.
Physical meaning
Scaling swaps each column's own units (years, dollars, grams) for one shared unit: "how unusual is this?". A z of 2 means "quite unusual" whether it is a height, a price or a temperature, so no column wins an argument just because its unit is small.

5. Worked example

Made-up numbers. A clinic table of 2,000 patients has: age (40 blanks), weight in kg (3 values of 0, 1 of 700), blood pressure, and smoker (yes/no, 120 blanks).

  1. Weight 0 and 700: errors. Set them to missing.
  2. Age: 2 % missing, no pattern found. Fill with the training-set median (52) and add age_missing.
  3. Smoker: 6 % missing, and the blanks are mostly from one clinic that didn’t ask. Add a third category “unknown” rather than guessing “no”.
  4. Scaling: standardise age, weight and blood pressure using training means and standard deviations, because the next step uses logistic regression.

6. Practice

  1. Standardise 130 when the mean is 100 and the standard deviation is 20.
  2. Should you delete the 50 most unusual credit card payments before training a fraud model?
  3. Why compute the fill value from training rows only?
Answers
  1. z = (130 − 100) / 20 = 1.5.
  2. No. Unusual payments are exactly what the model must learn about. Check them for errors, but keep the real ones.
  3. Because the test set must stand in for future data the model has never seen. Using it to compute fills lets its information leak into training.

Try it on your own data: the CSV profiler counts blanks per column and flags outliers.

New words: missing at random, imputation (filling gaps), missing flag, capping or winsorising, standardisation, z-score, min-max scaling, leakage.

  • Ask why a value is missing; fill with the median or most common value and add a “was missing” flag.
  • Fix outliers that are errors; keep real extremes, and never delete the cases you are trying to find.
  • Standardise (z = (x − mean) ÷ sd) or min-max scale for models that use distances or gradient descent; trees don’t need it.
  • Work out every fill value, mean and sd from the training rows only, to avoid leakage.

Check yourself

1Filling every missing value with the column mean does what to the spread?
2Age is in years (20 to 70) and income in dollars (20,000 to 150,000). Without scaling, a distance-based model mostly uses:
3What is the z-score of 85 if the mean is 70 and the standard deviation is 10?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.