All lessons

Module 3: Cleaning data · Lesson 3.2 · about 15 min

Duplicates

Find exact and partial duplicate rows, decide which copy to keep, and check that a key column really is unique before you join or count.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 15 minutes · uses Lesson 2.3. You’ll be able to find duplicates, remove the right copies and prove a key is unique.

Duplicates double-count: find exact repeats, repeats on the key columns, and check that every ID that should be unique really is.

In everyday terms: a guest list where someone signed in twice. Count heads from the list and you order an extra meal; the fix is to know which column (the name, the ticket number) should be unique.

1. Exact duplicates

Pythonorders_messy.csv

keep=False marks every copy, which is handy for looking at them side by side.

2. Duplicates on some columns

Rows can differ in an unimportant column but still be the same thing. Pass subset:

Python

Sort first, so the copy you want (the latest, the largest) is in the position keep picks.

3. Is the key unique?

Pythoncustomers.csv, orders.csv

order_id repeats because each row is an order line. That’s correct for this table, but it means “number of orders” is orders['order_id'].nunique(), not len(orders).

duplicated()
True for each row that repeats an earlier one.
duplicated(subset=[...])
Repeats only on chosen columns.
drop_duplicates(keep=...)
Remove repeats, keeping the first or last.
is_unique
True if a column has no repeats: the test for a key.
Physical meaning
A duplicate row makes one real event count twice: revenue, customers and averages all drift. A repeated key in a lookup table also multiplies rows when you join (Lesson 5.3).

“drop_duplicates() is always safe.” Two genuine orders of the same product by the same customer on the same day look identical if order_id is missing. Decide which columns define “the same thing” first.

Watch it happen, one step at a time:

The data.
order_idproductqty
50001P1012
50002P1051
50001P1012
50003P1012
Exact repeats. df.duplicated().sum() → 1 (row 2 repeats row 0).
Key check. df['order_id'].is_unique → False.
Look before deleting. Row 2 is a double export of 50001; row 3 is a different order, so keep it.
Remove. df.drop_duplicates() → 3 rows.

4. Your turn

Your turnorders_messy.csv
Your turnorders.csv
How it works

Principle: duplicated compares each row with all earlier rows (by hashing their values), so it’s fast even on millions of rows. Two rows count as the same only if every compared value matches exactly: 'Maple Ridge' and ' maple ridge' are different until you clean the text (next lessons). Clean first, then de-duplicate.

5. Practice

  1. Mark every copy of a duplicated email, not just the later ones.
  2. Keep each product’s cheapest offer from a table of offers.
  3. Why might drop_duplicates() find nothing in a messy file that clearly has repeats?
Answers
  1. df.duplicated(subset=['email'], keep=False).
  2. offers.sort_values('price').drop_duplicates('product_id', keep='first').
  3. The copies differ slightly (spaces, capitals, date formats, a timestamp column); clean those or use subset.

New words: duplicate, drop_duplicates, subset, keep, is_unique, key.

  • duplicated() flags repeats; .sum() counts them.
  • Use subset= to check the columns that define “the same thing”.
  • Check keys with is_unique before joining.
  • Look at duplicates before deleting them.

Check yourself

1What does df.duplicated() return?
2You want one row per customer, keeping their latest record. How?
3Why check that a key is unique before a join?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.