Home / Learn / Data & AI / ML foundations / Lesson 7.2
All lessons

Module 7: Judging models honestly · Lesson 7.2 · about 20 min

Cross-validation

One train/test split gives one lucky or unlucky score. k-fold cross-validation uses every row for testing once and gives k scores, an average and a spread. Run the folds yourself and learn the variants for groups and time.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 20 minutes · read first: 2.5, 7.1. You’ll be able to run k-fold cross-validation, report its result properly, and choose grouped or time-based folds when the data need them.

Instead of trusting one test, take turns: every slice of the data gets to be the test once, and you judge the model by all the scores together.

1. Why this matters

Almost every model choice (which model, which depth, which columns) is made by comparing validation scores. With one small validation set, the luck of which rows landed in it can be bigger than the difference you are trying to measure. It’s like judging a student on a single ten-question quiz: one bad day and the verdict is wrong. Cross-validation makes those comparisons trustworthy, and it appears in nearly every data-science job advert.

2. k-fold cross-validation

A fold is one slice of your rows. “k” is just the number of slices.

  1. Shuffle the rows and cut them into k equal folds (usually 5 or 10).
  2. For round i=1,…,k: train on all folds except fold i, then test on fold i.
  3. Report the average of the k scores and their standard deviation (how spread out they are).

Every row is tested exactly once, by a model that never saw it. And every row helps train k−1 of the models.

Watch it happen

Round 1. Cut the data into 5 folds. Train on folds 2 to 5, test on fold 1. Accuracy: 82 %.

Round 2. Now fold 2 sits out. A fresh model trains on folds 1, 3, 4 and 5 and scores 86 % on fold 2.

Round 3. Fold 3 is the test: 80 %. Notice how much one fold's score can move. A single split could have shown any of these.

Round 4. Fold 4 is the test: 84 %.

Round 5. Fold 5 is the test: 83 %. Every row has now been tested exactly once.

Sum up. Mean = (82 + 86 + 80 + 84 + 83) ÷ 5 = 83 %. Spread (sd) ≈ 2.2 points. Report "83 % ± 2.2", not just 83.

Try it: cross-validation, fold by fold
rounds done
mean accuracy
spread (sd)
Every row is used for testing exactly once and for training k − 1 times. One train/test split gives one score that depends on luck; k folds give k scores, so you see the average and how much it moves. Reshuffle and the single scores change a lot, the average much less. Five or ten folds are the usual choice.

Once the choice is made, retrain the chosen model on all the training data. A separate test set, kept out of the whole process, gives the final number (Lesson 2.5).

k
how many slices (folds) the data are cut into, and so how many models are trained
s₁ … sₖ
the score of each round, on the fold that model never saw
s̄ (mean)
the typical score: your best guess of how the model does on new data
sd
how much the score moves depending on which rows were tested
Physical meaning
It's a fair exam. A teacher splits the question bank into five quizzes so every question is asked once. A student who scores 83 ± 2 really knows the subject; a rival at 84 ± 3 is not clearly better.

3. Choosing the folds

Plain shuffled folds assume every row is a separate, independent case. When that isn’t true, the folds must follow the data:

  • Stratified folds keep the class balance the same in every fold. Important when one class is rare, like fraud (Lesson 4.5).
  • Grouped folds keep all rows of one group (a patient, a customer, a store) in the same fold, so the model is never tested on someone it has already met.
  • Time-series folds always train on the past and test on the period after it, moving forward through time, like a weather forecaster who can only use yesterday to predict tomorrow.
  • Leave-one-out (k=n, one row per fold): uses the most data, but is slow and its estimate can be noisy. 5 or 10 folds are usually better.

“Cross-validation replaces the test set.” Cross-validation is for choosing: comparing models and settings. Once you have picked the winner by its cross-validated score, that score is a little flattering, because you picked the best of several. The final, honest number still comes from a test set that was kept out of the whole process.

Maths and principle

Principle: estimate how a model trained on data like yours will do on new data, using every row once as new data.

The estimate: with fold scores s1,…,sk,

s‾=1k∑i=1ksi,sd=1k−1∑i=1k(si−s‾)2

Bigger k: each model trains on more data ((k−1)/k of it), so the estimate is less pessimistic, but the models overlap more and it takes k times the training. The folds share training rows, so the scores are not independent and the sd understates the true uncertainty a little.

Analogy: a teacher who splits the question bank into five quizzes and makes sure every question is in exactly one quiz, instead of judging a student on a single quiz.

Try it: fold accuracies 0.82, 0.86, 0.80, 0.84, 0.83. Mean and sd? (Mean 0.83. Deviations −0.01, 0.03, −0.03, 0.01, 0: squares add to 0.002; sd = √(0.002/4) ≈ 0.022.)

4. Worked example

Made-up numbers. Two models for predicting late deliveries, 5-fold cross-validation, accuracy in %:

fold 1 2 3 4 5 mean sd
logistic regression 81 84 79 83 82 81.8 1.9
random forest 85 86 83 87 85 85.2 1.5

The forest wins in every fold, by 3.4 points on average, about twice the spread. A single split could have shown fold 3’s forest (83) against fold 2’s logistic regression (84) and pointed the wrong way.

5. Practice

  1. In the simulation, reshuffle a few times. Which moves more: single fold scores or the mean?
  2. With 1,000 rows and 10 folds, how many rows does each model train on?
  3. Hospital data has several visits per patient. How should the folds be made?
Answers
  1. Single fold scores jump around; the mean moves much less.
  2. Grouped by patient, so all of a patient’s visits are in the same fold.

Try it on your own data: the small model trainer reports test accuracy on a held-out part of your CSV; change its random seed a few times and watch how much one split’s score moves.

New words: cross-validation, fold, k-fold, stratified, grouped folds, time-series split, leave-one-out.

  • k-fold cross-validation cuts the data into k folds; each fold is the test set once, so every row is tested by a model that never saw it.
  • Report the mean and the spread of the k scores; differences smaller than the spread may be luck.
  • Use stratified folds for rare classes, grouped folds when rows come in groups, and time-ordered folds for data in time.
  • Cross-validation is for choosing; the final score still comes from an untouched test set.

Check yourself

1In 5-fold cross-validation, each row is used for testing:
2Why report the spread of the k scores, not just the mean?
3Your data are daily sales over three years. Which validation is right?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.