Module 4: Grouping and aggregating · Lesson 4.3 · about 20 min
transform and filter: group results on every row
Use transform to put a group total or average back on every row (for shares, differences from the group mean and ranks within groups), and groupby filter to keep whole groups that pass a test.
Module 4 · about 20 minutes · uses Lesson 4.2. You’ll be able to compare each row with its group, and keep only the groups you want.
In everyday terms: a teacher writes each class’s average on every pupil’s report (transform), so each pupil can be compared with their class; then sends a note only to classes with more than 30 pupils (filter).
1. transform: a group value on every row
agg shrinks the table to one row per group; transform keeps every row and adds the group’s value to each. That’s what you need for shares, gaps from the average, and filling missing values per group (Lesson 3.1).
2. Ranks and running totals within groups
3. filter: keep whole groups
The same with transform, often faster: orders[orders.groupby('customer_id')['order_id'].transform('nunique') >= 10].
- groupby(k)[c].transform('sum')
- Same length as the table: each row gets its group's sum.
- x / transform('sum')
- Each row's share of its group.
- groupby(k).filter(lambda g: len(g) >= 10)
- Keep all rows of groups that pass.
- cumsum(), rank() per group
- Running totals and ranks that restart in each group.
- Physical meaning
- transform keeps the row grain while adding group context, which is what "share of store", "difference from the customer's average" and "fill with the group median" all need.
“agg and transform do the same thing.” agg gives one row per group; transform gives one value per original row. Use transform when the answer has to sit next to each row.
Watch it happen, one step at a time:
| store | qty |
|---|---|
| S01 | 2 |
| S01 | 6 |
| WEB | 4 |
| WEB | 4 |
| store | qty | store_total |
|---|---|---|
| S01 | 2 | 8 |
| S01 | 6 | 8 |
| WEB | 4 | 8 |
| WEB | 4 | 8 |
4. Your turn
How it works
Principle: agg maps each group to one value; transform maps each group to a value per row (broadcast back to the group’s rows, in the original order); filter maps each group to True/False and keeps the rows of the True groups. Same split, different combine.
| Output rows | Typical use | |
|---|---|---|
agg |
one per group | summary tables |
transform |
same as input | shares, gaps, group fills, ranks |
filter |
rows of passing groups | “customers with 5+ orders” |
5. Practice
- Each product’s share of its category’s units sold.
- Number each customer’s orders 1, 2, 3 in date order.
- Keep stores whose average discount is above 5 %.
Answers
- Sum units per product, then
df['units'] / df.groupby('category')['units'].transform('sum'). orders.sort_values('date').groupby('customer_id')['date'].rank(method='dense')(orcumcount() + 1on one row per order).orders.groupby('store_id').filter(lambda g: g['discount'].mean() > 0.05).
New words: transform, broadcast, group filter, cumsum, rank within group.
- transform returns one value per row, aligned with the table.
- Use it for shares, differences from group means and group-based fills.
- filter keeps or drops whole groups.
- cumsum, cumcount and rank on a groupby restart per group.
Check yourself
Sources
- pandas documentation, User Guide: Group by (transformation, filtration) https://pandas.pydata.org/docs/user_guide/groupby.html#transformation
- McKinney, W. (2022), Python for Data Analysis, 3rd edition, O'Reilly (open access) https://wesmckinney.com/book/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.