Module 4: Classification and logistic regression · Lesson 4.8 · about 25 min
Naive Bayes: counting words to catch spam
Bayes' rule turns "how often does this word appear in spam?" into "how likely is this email spam?". Naive Bayes multiplies those clues for every word. Type a message into a tiny spam filter and follow the sums.
Module 4 · about 25 minutes · read first: 1.6. You’ll be able to apply Bayes’ rule, build a naive Bayes text classifier by hand, and explain smoothing and why it works in log space.
1. Why this matters
Naive Bayes powered the first spam filters that really worked, in the early 2000s, and it is still a fast, strong first try for text: sorting support tickets, tagging articles, spotting a language. It needs very little data and learns just by counting. It is also the clearest example of Bayes’ rule, the logic of updating a belief when new evidence arrives, which runs right through statistics and machine learning.
2. Bayes’ rule
You know how often a word turns up in spam: , read “the probability of the word, given spam”. You want the reverse: how likely an email is spam, given that it contains the word. Bayes’ rule turns one into the other:
- P(spam)
- The prior: the share of all email that is spam, before reading a single word.
- P(word | spam)
- The likelihood: how typical the word is of spam.
- P(spam | word)
- The posterior: your updated belief after seeing the word.
- P(word)
- How common the word is overall. It just makes the answers for spam and not-spam add up to 1.
- Physical meaning
- A detective adding clues. You start with a hunch (the prior). Each clue that fits spam better than normal mail shifts the odds towards spam; each clue that fits normal mail shifts them back.
3. Many words: the naive step
An email has many words. Assume, naively, that each word appears independently of the others once we know the class. Then the likelihoods simply multiply (the symbol means “is proportional to”, and means “multiply all of these”):
Do the same for “normal”, then scale the two answers so they add to 1.
Start with the prior. 40 % of training emails were spam: P(spam) = 0.4, P(normal) = 0.6.
First clue: "prize". It is 14 times more common in spam (0.0150 against 0.00105). Spam score: 0.4 × 0.0150 = 0.0060. Normal: 0.6 × 0.00105 = 0.00063.
Second clue: "meeting". It is 7.6 times more common in normal mail (0.0163 against 0.00214). Spam: 0.0060 × 0.00214 = 1.29 × 10⁻⁵. Normal: 0.00063 × 0.0163 = 1.03 × 10⁻⁵.
Scale to add to 1. P(spam) = 1.29 ÷ (1.29 + 1.03) ≈ 0.56. The two clues nearly cancel: slightly more likely spam than not.
The independence assumption is plainly false (“free” makes “prize” more likely), so the probabilities come out too extreme. But the ranking of spam over normal is usually right, and that is what classification needs.
Nothing you type leaves this page. The filter learned only from the 12 example messages below.
| word | times in spam | times in normal | leans |
|---|
The 12 training messages
- spam: win a free prize now
- spam: claim your free cash prize
- spam: cheap meds free delivery now
- spam: you won cash click now
- spam: free entry win big today
- spam: limited offer click to claim
- normal: are we still meeting today
- normal: here are the notes from class
- normal: can you send the report now
- normal: lunch today at noon
- normal: your delivery arrives tomorrow
- normal: thanks for the notes see you in class
4. Two practical fixes
- Smoothing. If “invoice” never appeared in the spam examples, , and that one word would veto every other clue. Add 1 to every count (Laplace or add-one smoothing), so unseen words get a small chance instead of none.
- Logs. Multiplying hundreds of tiny probabilities gives a number too small for a computer to store, and it rounds to 0. Add their logarithms instead: . The class with the bigger total wins.
“P(word | spam) and P(spam | word) are the same thing.” They can be wildly different. Almost every spam email contains the word “the”, so P(“the” | spam) is near 1. But seeing “the” tells you almost nothing about whether an email is spam, because normal emails are full of it too. Bayes’ rule is exactly the tool for turning one into the other correctly.
Maths and principle
Principle: Bayes’ rule updates a prior belief with evidence; naive Bayes assumes the pieces of evidence are independent given the class.
Word probabilities with smoothing: if spam messages contain words in total, the word appears times, and the vocabulary has different words,
The decision in log space:
Try it: 1 % of patients have a disease. A test catches 90 % of cases and wrongly flags 5 % of healthy people. Someone tests positive. Chance they have it? (.)
5. Worked example
Made-up numbers. Training set: 40 spam emails with 400 words in total, 60 normal emails with 900 words; vocabulary of 1,000 different words. “Prize” appears 20 times in spam, 1 time in normal; “meeting” 2 times in spam, 30 in normal.
- Priors: , .
- ; .
- ; .
For the email “prize meeting”:
- Spam: 0.4 × 0.0150 × 0.00214 = 1.29 × 10⁻⁵
- Normal: 0.6 × 0.00105 × 0.0163 = 1.03 × 10⁻⁵
- . The two clues nearly cancel: a coin flip, slightly towards spam.
6. Practice
- In the simulation, which single word pushes a message hardest towards spam?
- Why do we work with sums of logs rather than products?
- A word appears in every email of both classes equally often. How does it affect the decision?
Answers
- Look for the largest “spam ×” ratio in the table; words such as “free”, “claim” and “prize” appear only in spam examples.
- Products of many small numbers become too small for the computer and round to zero; sums of logs stay in a safe range and give the same winner.
- Not at all: it multiplies both classes by the same amount. Such words (“the”, “and”) are often removed as “stop words”.
- Bayes’ rule turns P(evidence | class) and the prior P(class) into P(class | evidence).
- Naive Bayes multiplies the clues from every word, as if words were independent given the class.
- Add-one (Laplace) smoothing stops an unseen word from vetoing everything.
- Real filters add logarithms instead of multiplying, to avoid numbers too small to store.
- The probabilities are often too extreme, but the ranking is usually right.
Try it on your own data: the topic modelling (LDA) counts words across your own texts, the first step of any text classifier.
New words: Bayes’ rule, prior, likelihood, posterior, naive Bayes, conditional independence, Laplace smoothing, log probabilities, stop words.
Check yourself
Sources
- Manning, Raghavan and Schütze (2008), Introduction to Information Retrieval, Cambridge University Press, Chapter 13 (Text classification and naive Bayes) https://nlp.stanford.edu/IR-book/
- Sahami, Dumais, Heckerman and Horvitz (1998), A Bayesian approach to filtering junk e-mail, AAAI Workshop on Learning for Text Categorization
- scikit-learn user guide: Naive Bayes https://scikit-learn.org/stable/modules/naive_bayes.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.