All lessons
Machine learning glossary
173 words from the course, in plain English. "Also called" lists the other names you will meet in apps, papers and job ads. Each term links to the lesson that explains it.
No term matches. Try a shorter word.
A
- A/B test
- Also called: randomised experiment. Splitting people at random into groups that get different versions, so a difference in results can be blamed on the change. Lesson 2.3
- Activation function
- Also called: non-linearity. The bend a neuron applies to its weighted sum, such as ReLU, sigmoid or tanh. Without it, stacked layers collapse into one straight-line model. Lesson 9.2
- Adam
- A popular optimiser that keeps a running average of each weight's gradients and scales its step by how large they have been. Lesson 9.5
- Agglomerative clustering
- Also called: hierarchical clustering. Start with every point alone and keep merging the two closest clusters; the merges form a tree. Lesson 8.3
- Anomaly
- Also called: outlier. A row that does not fit the usual pattern: a fraud, a fault, a typo, or a genuine rare event. Lesson 8.5
- Attention
- Also called: self-attention. Each token scores every other token (query · key), softmaxes the scores and takes a weighted mix of their values. The core of the transformer. Lesson 10.4
- Attention head
- Also called: head. One of several attention calculations run side by side, each with its own weights, free to track a different relation. Lesson 10.4
- AUC
- Also called: area under the ROC curve. The chance that the model scores a random positive above a random negative. 0.5 is guessing, 1 is perfect. Lesson 4.6
B
- Backpropagation
- Also called: backprop. Getting every weight's gradient in one backward pass, by applying the chain rule from the loss back through the layers. Lesson 9.4
- Bag of words
- Also called: count vector. A document turned into a row of word counts, ignoring word order. Lesson 10.1
- Bagging
- Also called: bootstrap aggregating. Training many models on bootstrap samples and averaging (or voting) their answers to cut variance. Lesson 6.2
- Base model
- Also called: pretrained model, foundation model. An LLM after pretraining only: it continues text rather than answering requests. Lesson 10.6
- Baseline
- The simplest sensible prediction (the average, the most common class, last week's value). A model is only useful if it clearly beats it. Lesson 1.7
- Batch
- Also called: mini-batch. A small group of training rows used for one gradient step. Lesson 3.3
- Bayes' rule
- Also called: Bayes' theorem. Turns P(evidence | cause) into P(cause | evidence) using the prior: posterior ∝ likelihood × prior. Lesson 4.8
- Bias (error)
- Error from a model too simple to capture the pattern: it misses in the same way however much data it gets. Lesson 7.1
- Bias (parameter)
- Also called: intercept. The constant added to a weighted sum, so the line or neuron need not pass through zero. Lesson 1.3
- Bigram
- Also called: n-gram. A pair of words in a row. An n-gram model predicts the next word from the previous n − 1 by counting. Lesson 10.3
- Boosting
- Building models one after another, each fixing the mistakes of those before it. Lesson 6.5
- Bootstrap sample
- Also called: resample. A sample of n rows drawn with replacement from n rows: about 63 % of the rows appear, some more than once. Lesson 6.2
- Box plot
- Also called: box-and-whisker plot. A chart of the median, quartiles and outliers of a column. Lesson 2.2
C
- Causal mask
- In attention, hiding the tokens after each position, so a model writing text sees only what came before. Lesson 10.4
- Centroid
- Also called: cluster centre. The average point of a cluster. Lesson 8.1
- Chain rule
- The slope of a chain of steps is the product of each step's slope. Backpropagation is the chain rule applied efficiently. Lesson 1.5
- Classification
- Predicting a category, such as spam or not, or which of several products a customer buys. Lesson 4.1
- Clustering
- Grouping rows that are similar, with no answer column to learn from. Lesson 8.1
- CNN
- Also called: convolutional neural network. A network whose layers slide small shared filters over an image (or sound or signal) to find patterns anywhere. Lesson 9.6
- Confounder
- Also called: lurking variable. A third factor that drives two things and makes them look related. Lesson 2.3
- Confusion matrix
- A 2 × 2 table (or larger) counting true and false positives and negatives. Lesson 4.5
- Context window
- Also called: context length. How many tokens an LLM can see at once, prompt and answer together. Lesson 10.7
- Convolution
- Also called: filter, kernel. Sliding a small grid of weights over a picture and adding up the products at each place, giving a feature map. Lesson 9.6
- Correlation
- Also called: Pearson's r. How closely two columns move together along a straight line, from −1 to 1. Says nothing on its own about cause. Lesson 2.3
- Cosine similarity
- The cosine of the angle between two vectors: 1 for the same direction, 0 for unrelated. Used to compare documents and embeddings. Lesson 1.4
- Cross-entropy
- Also called: log-loss, cross-entropy loss. The average of −ln(probability given to the true answer). The loss for classifiers and language models. Lesson 4.3
- Cross-validation
- Also called: k-fold cross-validation. Splitting the training data into k folds, training on k − 1 and scoring on the one left out, k times, and averaging. Lesson 7.2
- Curse of dimensionality
- In many dimensions every point is far from every other, so distances and neighbours stop meaning much. Lesson 4.7
D
- Data leakage
- Also called: leakage, target leakage. Information the model would not have at prediction time sneaking into training, making test scores too good to be true. Lesson 7.4
- Decision boundary
- The line or surface where a classifier switches from one class to the other. Lesson 4.4
- Decision tree
- A model that asks a sequence of yes/no questions about the columns and predicts from the leaf it ends in. Lesson 5.1
- Dendrogram
- The tree diagram of hierarchical clustering; cutting it at a height gives clusters. Lesson 8.3
- Derivative
- Also called: slope. How fast a function changes as its input changes: the slope of the tangent at a point. Lesson 1.5
- Dimensionality reduction
- Describing rows with fewer numbers while keeping most of the information, as PCA does. Lesson 8.4
- Dot product
- Also called: inner product. Multiply two vectors number by number and add up. A weighted sum is a dot product. Lesson 1.4
- Dropout
- Switching off a random share of neurons at each training step, so no neuron can rely on another; a defence against overfitting. Lesson 9.5
E
- Early stopping
- Stopping training when validation loss stops improving, and keeping the best weights. Lesson 3.7
- Embedding
- Also called: vector representation. A learned list of numbers for a word, product or item, placed so similar ones are close. Lesson 10.2
- Ensemble
- Several models combined by voting or averaging, usually more accurate than any one. Lesson 6.1
- Entropy
- A measure of mixture or surprise in bits: 0 for a pure group, 1 for a 50/50 split of two classes. Lesson 5.2
- Epoch
- One pass over all the training rows. Lesson 3.3
F
- F1 score
- The harmonic mean of precision and recall: high only when both are high. Lesson 4.5
- Feature
- Also called: column, input, predictor, variable. A measured property used to make a prediction, such as size or age. Lesson 1.2
- Feature engineering
- Making new columns from old ones (squares, ratios, logs, dates) so a model can use the pattern. Lesson 3.6
- Feature importance
- How much each column contributes to a model's predictions. Permutation importance is the more trustworthy kind. Lesson 6.4
- Feature map
- The grid of filter scores a convolution produces; also, more broadly, a set of learned features. Lesson 9.6
- Feed-forward network
- Also called: MLP, fully connected network. Layers of neurons where each feeds the next; in a transformer, the part that processes each token on its own. Lesson 10.5
- Fine-tuning
- Also called: supervised fine-tuning, SFT. Training an existing model further on a smaller set of examples for a new task or style. Lesson 10.6
- Fold
- One of the k equal parts the data is split into for cross-validation. Lesson 7.2
G
- Generalisation
- How well a model does on new data it never saw in training. The point of the whole exercise. Lesson 2.5
- Gini impurity
- The chance two random rows from a group have different classes: 0 for pure, 0.5 for a 50/50 split. Lesson 5.2
- Gradient
- The list of slopes of the loss with respect to every weight; it points uphill. Lesson 1.5
- Gradient boosting
- Also called: XGBoost, LightGBM, CatBoost. Boosting where each new tree is fitted to the residuals (the negative gradient of the loss) of the model so far. Lesson 6.6
- Gradient descent
- Repeatedly stepping each weight a little downhill against its gradient, to lower the loss. Lesson 3.3
H
- Hallucination
- Also called: confabulation. An LLM stating something false in fluent, confident text. Lesson 10.7
- Histogram
- A bar chart of how many values fall in each range of a column. Lesson 2.2
- Hyperparameter
- Also called: setting. A choice made before training (depth, learning rate, k), tuned with validation data rather than learned. Lesson 7.3
I
- IDF
- Also called: inverse document frequency. ln(number of documents ÷ number containing the word): large for rare words, 0 for words in every document. Lesson 10.1
- Imputation
- Filling missing values with a sensible guess, such as the median, learned from training data only. Lesson 2.4
- Inertia
- Also called: within-cluster sum of squares. The total squared distance from each point to its own cluster's centre. k-means makes it as small as it can. Lesson 8.1
- Information gain
- How much a split lowers impurity, weighting each side by its size. Trees choose the split with the most. Lesson 5.3
- Instruction tuning
- Training a base LLM on example requests and good replies, so it answers instead of just continuing text. Lesson 10.6
- Intercept
- Also called: bias. The value of a straight line where the input is 0. Lesson 1.3
- Irreducible error
- Also called: noise. The part of the error no model can remove, because the outcome depends on things not in the data. Lesson 7.1
K
- k-means
- Clustering by repeating two steps: give each point to its nearest centre, then move each centre to the mean of its points. Lesson 8.1
- k-nearest neighbours
- Also called: k-NN. Predicting from the k most similar training rows: a vote for classes, an average for numbers. Lesson 4.7
- Key
- In attention, the vector a token offers to be matched against other tokens' queries. Lesson 10.4
- Knowledge cut-off
- Also called: training cut-off. The date after which an LLM's training text stops, so it knows nothing later on its own. Lesson 10.6
L
- Label
- Also called: target, answer, outcome. The value a supervised model learns to predict. Lesson 1.2
- Language model
- A model that gives a probability to every possible next token, given the text so far. Lesson 10.3
- Large language model
- Also called: LLM. A very large transformer language model, trained on huge amounts of text and then tuned to follow instructions. Lesson 10.7
- Lasso
- Also called: L1 regularisation. Regression with a penalty on the absolute size of the weights; it sets some weights to exactly zero. Lesson 3.8
- Layer normalisation
- Also called: layer norm. Rescaling each token's numbers to mean 0 and spread 1, to keep a deep network's numbers in a steady range. Lesson 10.5
- Learning curve
- A plot of training and validation error against the amount of training data. Lesson 7.1
- Learning rate
- Also called: step size, η. How far each gradient step moves the weights. Too small crawls; too big overshoots. Lesson 3.3
- Least squares
- Also called: ordinary least squares, OLS. Choosing the line that makes the sum of squared errors as small as possible. Lesson 3.2
- Linear model
- Also called: linear regression. A prediction made as a weighted sum of the inputs plus a bias: a line, plane or hyperplane. Lesson 3.1
- Linkage
- How hierarchical clustering measures the distance between two clusters: closest pair, farthest pair, or average. Lesson 8.3
- Log-odds
- Also called: logit. ln(p ÷ (1 − p)). Logistic regression is a linear model of the log-odds. Lesson 4.2
- Logistic regression
- A linear model whose output goes through a sigmoid to give a probability of the positive class. Lesson 4.2
- Logits
- Also called: scores. A model's raw scores before softmax turns them into probabilities. Lesson 10.5
- LoRA
- Also called: low-rank adaptation, adapter. A cheap way to fine-tune a large model by training small extra weight matrices alongside the frozen original. Lesson 10.6
- Loss
- Also called: cost, error, objective. One number saying how wrong the model is on the data. Training makes it small. Lesson 3.1
- LSTM
- Also called: long short-term memory, GRU. A recurrent cell with gates that decide what to keep and forget, so it remembers over many steps. Lesson 9.7
M
- MAE
- Also called: mean absolute error. The average size of the errors, in the units of the answer. Lesson 3.5
- Mean
- Also called: average. The sum of the values divided by how many there are. Lesson 1.6
- Median
- The middle value when sorted. Unlike the mean, it barely moves for a few extreme values. Lesson 1.6
- Missing values
- Also called: NA, null. Empty cells. Why they are missing matters as much as how many there are. Lesson 2.4
- Model
- A rule with adjustable numbers that turns inputs into a prediction. Lesson 1.1
- Momentum
- Keeping a running average of recent gradient steps so training rolls through small bumps. Lesson 9.5
- MSE
- Also called: mean squared error. The average of the squared errors. RMSE is its square root, in the answer's units. Lesson 3.1
- Multicollinearity
- Inputs so closely related that a linear model cannot tell their effects apart, making weights unstable. Lesson 3.4
N
- Naive Bayes
- A classifier that multiplies each feature's probability as if they were independent. Fast and good for text. Lesson 4.8
- Neuron
- Also called: unit, node, perceptron. A weighted sum of inputs plus a bias, passed through an activation function. Lesson 9.1
- Normal distribution
- Also called: bell curve, Gaussian. The symmetric bell-shaped spread of values described by a mean and a standard deviation. Lesson 1.6
- Normal equation
- The formula w = (XᵀX)⁻¹Xᵀy that gives the least-squares weights in one step. Lesson 3.4
O
- Odds
- p ÷ (1 − p): a 75 % chance is odds of 3 to 1. Lesson 4.2
- One-hot encoding
- Also called: dummy variables. Turning a category into 0/1 columns, one per category. Lesson 1.2
- Out-of-bag score
- Also called: OOB. A random forest's accuracy on the rows each tree did not see, a free estimate of test performance. Lesson 6.3
- Overfitting
- Learning the noise of the training data, so training scores are great and new data scores are poor. Lesson 3.7
P
- Parameter
- Also called: weight, coefficient. A number inside a model that training adjusts. LLM sizes count these. Lesson 1.1
- PCA
- Also called: principal component analysis. Finding the directions of greatest spread in the data, so a few numbers describe each row. Lesson 8.4
- Perceptron
- The first learning neuron (1958): a weighted vote with a step, trained by nudging weights after each mistake. Lesson 9.1
- Permutation importance
- How much a model's score drops when one column is shuffled on held-out data. Lesson 6.4
- Perplexity
- e to the power of a language model's average loss: how many tokens it is, in effect, choosing between. Lesson 10.3
- Pooling
- Also called: max pooling. Shrinking a feature map by keeping the largest (or average) value in each small square. Lesson 9.6
- Positional encoding
- Also called: position embedding. A signal added to each token so a transformer knows word order. Lesson 10.5
- Precision
- Also called: positive predictive value. Of the cases the model flagged, the share that were truly positive. Lesson 4.5
- Pretraining
- An LLM's first and largest training stage: predicting the next token over trillions of tokens of text. Lesson 10.6
- Prior
- What you believed before seeing the evidence, such as the share of all mail that is spam. Lesson 4.8
- Probability
- A number from 0 to 1 for how likely something is. Lesson 1.6
- Prompt
- Also called: system prompt. All the text given to an LLM: hidden instructions, the conversation, documents and the user's message. Lesson 10.7
- Pruning
- Limiting or cutting back a decision tree so it does not memorise the training data. Lesson 5.5
Q
- Query
- In attention, the vector a token uses to ask what it is looking for. Lesson 10.4
R
- R²
- Also called: R-squared, coefficient of determination. The share of the answer's variance a model explains: 1 is perfect, 0 is no better than the average. Lesson 3.5
- RAG
- Also called: retrieval-augmented generation. Finding relevant passages (often with embeddings) and putting them in an LLM's prompt so it answers from them. Lesson 10.7
- Random forest
- Many decision trees, each grown on a bootstrap sample with a random subset of columns at each split, voting together. Lesson 6.3
- Recall
- Also called: sensitivity, true positive rate. Of the truly positive cases, the share the model caught. Lesson 4.5
- Recurrent neural network
- Also called: RNN. A network that reads a sequence one item at a time, updating a hidden state with the same weights each step. Lesson 9.7
- Regression
- Predicting a number, such as a price or a time. Lesson 1.2
- Regression tree
- A decision tree that predicts a number: the mean of the training rows in each leaf. Lesson 5.4
- Regularisation
- Adding a penalty for large weights (or otherwise limiting a model) to reduce overfitting. Lesson 3.8
- ReLU
- Also called: rectified linear unit. The activation max(0, z): the default in deep networks because its slope does not fade. Lesson 9.2
- Residual
- Also called: error. The true value minus the prediction for one row. Lesson 3.1
- Residual connection
- Also called: skip connection. Adding a layer's input to its output, so information and gradients can skip past it. Lesson 10.5
- Ridge regression
- Also called: L2 regularisation, weight decay. Regression with a penalty on the squared size of the weights, shrinking them towards zero. Lesson 3.8
- RLHF
- Also called: reinforcement learning from human feedback, DPO. Tuning an LLM towards the answers people prefer, learned from comparisons of two answers. Lesson 10.6
- RMSE
- Also called: root mean squared error. The square root of the mean squared error, in the answer's units; punishes big misses more than MAE. Lesson 3.5
- ROC curve
- True positive rate against false positive rate, for every possible threshold. Lesson 4.6
S
- Scaling
- Also called: standardisation, z-scores, min-max. Putting columns on comparable scales, so distances and gradient descent treat them fairly. Lesson 2.4
- Self-supervised learning
- Learning from data that supplies its own labels, such as the next word in a text. Lesson 10.6
- Sigmoid
- Also called: logistic function. The S-shaped curve 1 ÷ (1 + e^−z) that squashes any number into a probability between 0 and 1. Lesson 4.2
- Silhouette score
- How much closer each point is to its own cluster than to the next one, from −1 to 1. Lesson 8.2
- Simpson's paradox
- A trend that appears in every group but reverses when the groups are combined, or the other way round. Lesson 2.3
- Softmax
- Turning a list of scores into probabilities that add up to 1, with the biggest score getting the most. Lesson 9.3
- Specificity
- Also called: true negative rate. Of the truly negative cases, the share the model correctly left alone. Lesson 4.5
- Standard deviation
- Also called: SD, σ. The typical distance of values from their mean: the square root of the variance. Lesson 1.6
- Standard error
- Also called: SE. How much an estimate such as a mean would vary between samples: the standard deviation ÷ √n. Lesson 1.6
- Stochastic gradient descent
- Also called: SGD. Gradient descent using a small random batch of rows for each step instead of all of them. Lesson 3.3
- Stratified split
- A split that keeps each class's share the same in every part. Lesson 2.5
- Supervised learning
- Learning from examples that include the right answer. Lesson 1.2
T
- Temperature
- A setting that sharpens (below 1) or flattens (above 1) a model's probabilities before a token is drawn. Lesson 10.3
- Test set
- Also called: hold-out set. Data set aside and used once, at the end, to estimate performance on new data. Lesson 2.5
- TF-IDF
- Also called: term frequency-inverse document frequency. A word's count in a document times its IDF: high for words frequent here and rare elsewhere. Lesson 10.1
- Threshold
- Also called: cut-off. The probability above which a classifier says "yes". Moving it trades precision against recall. Lesson 4.1
- Token
- The unit a language model reads and writes: a word or a piece of a word. Lesson 10.3
- Training
- Also called: fitting, learning. Adjusting a model's parameters so its predictions match the training data better. Lesson 1.1
- Transformer
- A network built from stacked blocks of attention and feed-forward layers; the design behind LLMs. Lesson 10.5
U
- Underfitting
- A model too simple to capture the pattern, so it does poorly even on its training data. Lesson 3.7
- Universal approximation
- One hidden layer with enough neurons can approximate any reasonable function; it says nothing about finding it. Lesson 9.3
- Unsupervised learning
- Finding structure (groups, directions, oddities) in data with no answer column. Lesson 8.1
V
- Validation set
- Also called: dev set. Data held back from training to choose settings and decide when to stop. Lesson 2.5
- Value
- In attention, the vector a token passes on to the tokens that attend to it. Lesson 10.4
- Vanishing gradient
- Also called: exploding gradient. Gradients shrinking (or growing) as they pass back through many layers or time steps, so early weights barely learn. Lesson 9.2
- Variance (error)
- Error from a model so flexible it changes a lot with each training sample. Lesson 7.1
- Variance (statistics)
- The average squared distance of values from their mean. Lesson 1.6
- Vector
- An ordered list of numbers, such as one row of features or a word's embedding. Lesson 1.4
W
- Weak learner
- Also called: stump. A model only a little better than guessing, such as a one-split tree; boosting combines many. Lesson 6.5
- Weight
- Also called: coefficient. How much an input counts in a weighted sum. Lesson 1.3
- Wisdom of crowds
- Many independent, decent guesses averaged are usually better than any one, the idea behind ensembles. Lesson 6.1
- Word2vec
- A 2013 method that learns word embeddings by predicting each word's neighbours in a large text. Lesson 10.2
X
- XOR problem
- Four points no single straight line can separate; solving it needs a hidden layer or extra features. Lesson 4.4
Z
- z-score
- Also called: standard score. How many standard deviations a value is from the mean. Lesson 2.4