Module 3: Cleaning data · Lesson 3.3 · about 20 min
Data types and converting them
Check each column's type and fix the wrong ones: text to numbers with to_numeric (stripping $ and %), numbers to text for codes, categories for repeated labels, and booleans.
Module 3 · about 20 minutes · uses Lesson 1.6. You’ll be able to fix columns that were read with the wrong type.
In everyday terms: numbers stored as text are like prices written on paper tags: you can read them, but the till cannot add them. Converting the type is typing them into the till.
1. Spot the wrong types
Unit Price and Discount should be numbers but are text (str), because some cells contain $ or %. You can’t sum or average them until they’re converted.
2. Text to numbers
If some values can’t be read, errors='coerce' turns them into NaN, and you can see which:
3. Numbers to text, categories and booleans
astype(str)for codes you’ll never do maths on (ids, postcodes, phone numbers).- An ordered category sorts in a meaningful order and saves memory on repeated labels.
mapwith a dictionary turns labels into True/False (or any values).astype(int)fails if the column has NaN; useastype('Int64')(capital I), which allows missing whole numbers.
- int64 / float64
- Whole numbers / decimals.
- object or string
- Text.
- datetime64
- Dates and times.
- category
- Text with few repeated values, stored compactly.
- pd.to_numeric(errors='coerce')
- Convert text to numbers; anything unreadable becomes NaN for you to inspect.
- Physical meaning
- The type decides what operations mean. "10" + "5" is "105" as text and 15 as numbers; "10" sorts before "9" as text. Fix types right after reading.
“If it looks like a number, pandas treats it as one.” One ”$” or “n/a” in a column turns the whole column into text. info() shows object instead of float64: that is your clue.
Watch it happen, one step at a time:
.str.replace('$', '') → ['4.50', '12', 'n/a', '7.25'].pd.to_numeric(..., errors='coerce') → 4.5, 12.0, NaN, 7.25 (float64).4. Your turn
How it works
Principle: a pandas column holds one type. If even one value can’t be read as a number, read_csv keeps the whole column as text. Converting is deliberate: you decide how to treat the odd values instead of pandas guessing.
Types you’ll meet: int64, float64 (numbers), bool, str (text, pandas 3), datetime64[ns] (dates, next lessons), category, and the nullable Int64, boolean.
5. Practice
- Convert
'1,250.50'style text to numbers. - Why keep customer ids as text even if they look like numbers?
- What does
astype('Int64')allow thatastype(int)doesn’t?
Answers
pd.to_numeric(s.str.replace(',', '', regex=False)).- You never add them up, leading zeros matter, and some systems mix letters in later.
- Missing values (shown as
<NA>) in a whole-number column.
New words: dtype, to_numeric, coerce, astype, category, ordered category, nullable integer.
- Check dtypes with info() after reading.
- Clean symbols, then pd.to_numeric(errors=‘coerce’) and inspect new NaN.
- Use astype for safe conversions; category for repeated text; bool for flags.
- Text numbers add and sort as text: “10” + “5” = “105”.
Check yourself
Sources
- pandas documentation, User Guide: Essential basic functionality (dtypes) https://pandas.pydata.org/docs/user_guide/basics.html#dtypes
- McKinney, W. (2022), Python for Data Analysis, 3rd edition, O'Reilly (open access) https://wesmckinney.com/book/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.