Module 8: Analyses employers ask for · Lesson 8.8 · about 15 min
Fast pandas: vectorise, avoid loops
Make pandas code fast: whole-column (vectorised) operations instead of loops and apply, np.where and np.select for conditions, categories for repeated text, reading only the columns you need, and timing code to see the difference.
Module 8 · about 15 minutes · uses Modules 2 and 3. You’ll be able to spot slow pandas code and rewrite it.
In everyday terms: a supermarket checkout: scanning a full basket at once versus walking back to the shelf for each item.
1. Loop, apply, vectorised
Same result, very different speed: the vectorised line is usually hundreds of times faster. Times here are in your browser; on a laptop all three are faster, but the order is the same.
2. Conditions without loops
Rules of thumb:
- Arithmetic and comparisons on columns: always vectorised.
ifper row: usenp.whereornp.select.- A lookup:
mapormerge, never a loop. - Text: the
.strmethods. - Only when nothing else fits:
apply, and preferably on a groupby, not on every row.
3. Memory: categories and fewer columns
Read only the columns you need (usecols), and store repeated text as category. On a file with millions of rows this is the difference between “fits in memory” and “crashes”.
- Vectorised
- df.qty * df.price: one operation on the whole column, in compiled code.
- apply(axis=1)
- A Python function per row: convenient but slow.
- np.where / np.select
- If-then rules on whole columns.
- astype('category')
- Stores repeated text once; less memory, faster groupby.
- Physical meaning
- A loop pays Python's overhead on every row; a column operation pays it once and does the rest in C. On a million rows that is the difference between a blink and a coffee break.
“apply is vectorised.” It still calls your Python function once per row. Use it when no column operation exists, not by default.
Watch it happen, one step at a time:
for i, row in df.iterrows(): ... one row at a time.df['pay'] = np.where(df.qty >= 3, df.price * 0.9, df.price)| qty | price | pay |
|---|---|---|
| 1 | 10 | 10.0 |
| 3 | 10 | 9.0 |
| 5 | 20 | 18.0 |
4. Your turn
How it works
Principle: a pandas column is a NumPy array: numbers packed side by side in memory. A vectorised operation hands the whole array to compiled C code, which runs through it in one tight loop. A Python loop instead does a slow lookup, type check and object creation for every row:
When pandas is too slow (tens of millions of rows): Polars, DuckDB (SQL on files) or Spark (Data at Scale course) use the same ideas across many cores or machines.
5. Practice
- Replace
df.apply(lambda r: r.a + r.b, axis=1). - A loop builds a list of strings with
if/elif/else. What do you use instead? - Why does
usecolsspeed upread_csv?
Answers
df['a'] + df['b'].np.select([cond1, cond2], [val1, val2], default=val3).- pandas skips parsing the other columns and never stores them.
Try it on your own data: big-O plotter shows how loop costs grow with the number of rows.
New words: vectorised, iterrows, apply, np.where, category dtype, usecols.
- Prefer column operations over loops and apply.
- np.where for one rule, np.select for several.
- category dtype for repeated text.
- Read only the columns you need (usecols=).
Check yourself
Sources
- pandas documentation, User Guide: Enhancing performance https://pandas.pydata.org/docs/user_guide/enhancingperf.html
- pandas documentation, User Guide: Scaling to large datasets https://pandas.pydata.org/docs/user_guide/scale.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.