Home / Learn / Data & AI / ML foundations / Lesson 4.8
All lessons

Module 4: Classification and logistic regression · Lesson 4.8 · about 25 min

Naive Bayes: counting words to catch spam

Bayes' rule turns "how often does this word appear in spam?" into "how likely is this email spam?". Naive Bayes multiplies those clues for every word. Type a message into a tiny spam filter and follow the sums.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 25 minutes · read first: 1.6. You’ll be able to apply Bayes’ rule, build a naive Bayes text classifier by hand, and explain smoothing and why it works in log space.

Start with how common spam is, then let every word nudge that belief up or down, and add up the nudges.

1. Why this matters

Naive Bayes powered the first spam filters that really worked, in the early 2000s, and it is still a fast, strong first try for text: sorting support tickets, tagging articles, spotting a language. It needs very little data and learns just by counting. It is also the clearest example of Bayes’ rule, the logic of updating a belief when new evidence arrives, which runs right through statistics and machine learning.

2. Bayes’ rule

You know how often a word turns up in spam: P(word|spam), read “the probability of the word, given spam”. You want the reverse: how likely an email is spam, given that it contains the word. Bayes’ rule turns one into the other:

P(spam|word)=P(word|spam)P(spam)P(word)
P(spam)
The prior: the share of all email that is spam, before reading a single word.
P(word | spam)
The likelihood: how typical the word is of spam.
P(spam | word)
The posterior: your updated belief after seeing the word.
P(word)
How common the word is overall. It just makes the answers for spam and not-spam add up to 1.
Physical meaning
A detective adding clues. You start with a hunch (the prior). Each clue that fits spam better than normal mail shifts the odds towards spam; each clue that fits normal mail shifts them back.

3. Many words: the naive step

An email has many words. Assume, naively, that each word appears independently of the others once we know the class. Then the likelihoods simply multiply (the symbol ∝ means “is proportional to”, and ∏ means “multiply all of these”):

P(spam|w1,…,wm)∝P(spam)∏k=1mP(wk|spam)

Do the same for “normal”, then scale the two answers so they add to 1.

Start with the prior. 40 % of training emails were spam: P(spam) = 0.4, P(normal) = 0.6.

First clue: "prize". It is 14 times more common in spam (0.0150 against 0.00105). Spam score: 0.4 × 0.0150 = 0.0060. Normal: 0.6 × 0.00105 = 0.00063.

Second clue: "meeting". It is 7.6 times more common in normal mail (0.0163 against 0.00214). Spam: 0.0060 × 0.00214 = 1.29 × 10⁻⁵. Normal: 0.00063 × 0.0163 = 1.03 × 10⁻⁵.

Scale to add to 1. P(spam) = 1.29 ÷ (1.29 + 1.03) ≈ 0.56. The two clues nearly cancel: slightly more likely spam than not.

The independence assumption is plainly false (“free” makes “prize” more likely), so the probabilities come out too extreme. But the ranking of spam over normal is usually right, and that is what classification needs.

Try it: be the spam filter

Nothing you type leaves this page. The filter learned only from the 12 example messages below.

wordtimes in spamtimes in normalleans
P(spam | message)
verdict
The 12 training messages
  • spam: win a free prize now
  • spam: claim your free cash prize
  • spam: cheap meds free delivery now
  • spam: you won cash click now
  • spam: free entry win big today
  • spam: limited offer click to claim
  • normal: are we still meeting today
  • normal: here are the notes from class
  • normal: can you send the report now
  • normal: lunch today at noon
  • normal: your delivery arrives tomorrow
  • normal: thanks for the notes see you in class
Each word's chance in spam and in normal mail is counted from the examples, with 1 added to every count so an unseen word never makes a probability zero. "Leans spam ×3" means the word is 3 times as likely in spam as in normal mail. The filter multiplies the chances of all the words together, as if each word appeared independently of the others (the "naive" part, clearly untrue, yet it works well). Words it never saw are skipped.

4. Two practical fixes

  • Smoothing. If “invoice” never appeared in the spam examples, P(invoice|spam)=0, and that one word would veto every other clue. Add 1 to every count (Laplace or add-one smoothing), so unseen words get a small chance instead of none.
  • Logs. Multiplying hundreds of tiny probabilities gives a number too small for a computer to store, and it rounds to 0. Add their logarithms instead: ln⁡P(spam)+∑kln⁡P(wk|spam). The class with the bigger total wins.

“P(word | spam) and P(spam | word) are the same thing.” They can be wildly different. Almost every spam email contains the word “the”, so P(“the” | spam) is near 1. But seeing “the” tells you almost nothing about whether an email is spam, because normal emails are full of it too. Bayes’ rule is exactly the tool for turning one into the other correctly.

Maths and principle

Principle: Bayes’ rule updates a prior belief with evidence; naive Bayes assumes the pieces of evidence are independent given the class.

Word probabilities with smoothing: if spam messages contain Nspam words in total, the word w appears nw,spam times, and the vocabulary has V different words,

P(w|spam)=nw,spam+1Nspam+V

The decision in log space:

score(c)=ln⁡P(c)+∑kln⁡P(wk|c),yˆ=arg⁡maxc⁡score(c)

Try it: 1 % of patients have a disease. A test catches 90 % of cases and wrongly flags 5 % of healthy people. Someone tests positive. Chance they have it? (0.9×0.010.9×0.01+0.05×0.99=0.0090.0585≈15%.)

5. Worked example

Made-up numbers. Training set: 40 spam emails with 400 words in total, 60 normal emails with 900 words; vocabulary of 1,000 different words. “Prize” appears 20 times in spam, 1 time in normal; “meeting” 2 times in spam, 30 in normal.

  • Priors: P(spam)=0.4, P(normal)=0.6.
  • P(prize|spam)=21/1400=0.0150; P(prize|normal)=2/1900=0.00105.
  • P(meeting|spam)=3/1400=0.00214; P(meeting|normal)=31/1900=0.0163.

For the email “prize meeting”:

  • Spam: 0.4 × 0.0150 × 0.00214 = 1.29 × 10⁻⁵
  • Normal: 0.6 × 0.00105 × 0.0163 = 1.03 × 10⁻⁵
  • P(spam)=1.29/(1.29+1.03)≈0.56. The two clues nearly cancel: a coin flip, slightly towards spam.

6. Practice

  1. In the simulation, which single word pushes a message hardest towards spam?
  2. Why do we work with sums of logs rather than products?
  3. A word appears in every email of both classes equally often. How does it affect the decision?
Answers
  1. Look for the largest “spam ×” ratio in the table; words such as “free”, “claim” and “prize” appear only in spam examples.
  2. Products of many small numbers become too small for the computer and round to zero; sums of logs stay in a safe range and give the same winner.
  3. Not at all: it multiplies both classes by the same amount. Such words (“the”, “and”) are often removed as “stop words”.
  • Bayes’ rule turns P(evidence | class) and the prior P(class) into P(class | evidence).
  • Naive Bayes multiplies the clues from every word, as if words were independent given the class.
  • Add-one (Laplace) smoothing stops an unseen word from vetoing everything.
  • Real filters add logarithms instead of multiplying, to avoid numbers too small to store.
  • The probabilities are often too extreme, but the ranking is usually right.

Try it on your own data: the topic modelling (LDA) counts words across your own texts, the first step of any text classifier.

New words: Bayes’ rule, prior, likelihood, posterior, naive Bayes, conditional independence, Laplace smoothing, log probabilities, stop words.

Check yourself

1Bayes' rule gives P(spam | word) from:
2What is "naive" about naive Bayes?
3Why add 1 to every word count (Laplace smoothing)?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.