All lessons

Module 4: Grouping and aggregating · Lesson 4.3 · about 20 min

transform and filter: group results on every row

Use transform to put a group total or average back on every row (for shares, differences from the group mean and ranks within groups), and groupby filter to keep whole groups that pass a test.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 20 minutes · uses Lesson 4.2. You’ll be able to compare each row with its group, and keep only the groups you want.

transform puts a group's result back on every row of the group; filter keeps or drops whole groups by a rule.

In everyday terms: a teacher writes each class’s average on every pupil’s report (transform), so each pupil can be compared with their class; then sends a note only to classes with more than 30 pupils (filter).

1. transform: a group value on every row

Pythonorders.csv

agg shrinks the table to one row per group; transform keeps every row and adds the group’s value to each. That’s what you need for shares, gaps from the average, and filling missing values per group (Lesson 3.1).

2. Ranks and running totals within groups

Pythonorders.csv, products.csv

3. filter: keep whole groups

Pythonorders.csv

The same with transform, often faster: orders[orders.groupby('customer_id')['order_id'].transform('nunique') >= 10].

groupby(k)[c].transform('sum')
Same length as the table: each row gets its group's sum.
x / transform('sum')
Each row's share of its group.
groupby(k).filter(lambda g: len(g) >= 10)
Keep all rows of groups that pass.
cumsum(), rank() per group
Running totals and ranks that restart in each group.
Physical meaning
transform keeps the row grain while adding group context, which is what "share of store", "difference from the customer's average" and "fill with the group median" all need.

“agg and transform do the same thing.” agg gives one row per group; transform gives one value per original row. Use transform when the answer has to sit next to each row.

Watch it happen, one step at a time:

The rows.
storeqty
S012
S016
WEB4
WEB4
Group totals. S01 → 8, WEB → 8.
transform('sum') puts them back on each row.
storeqtystore_total
S0128
S0168
WEB48
WEB48
Share. qty ÷ store_total → 0.25, 0.75, 0.5, 0.5.

4. Your turn

Your turnproducts.csv
Your turnorders.csv
How it works

Principle: agg maps each group to one value; transform maps each group to a value per row (broadcast back to the group’s rows, in the original order); filter maps each group to True/False and keeps the rows of the True groups. Same split, different combine.

Output rows Typical use
agg one per group summary tables
transform same as input shares, gaps, group fills, ranks
filter rows of passing groups “customers with 5+ orders”

5. Practice

  1. Each product’s share of its category’s units sold.
  2. Number each customer’s orders 1, 2, 3 in date order.
  3. Keep stores whose average discount is above 5 %.
Answers
  1. Sum units per product, then df['units'] / df.groupby('category')['units'].transform('sum').
  2. orders.sort_values('date').groupby('customer_id')['date'].rank(method='dense') (or cumcount() + 1 on one row per order).
  3. orders.groupby('store_id').filter(lambda g: g['discount'].mean() > 0.05).

New words: transform, broadcast, group filter, cumsum, rank within group.

  • transform returns one value per row, aligned with the table.
  • Use it for shares, differences from group means and group-based fills.
  • filter keeps or drops whole groups.
  • cumsum, cumcount and rank on a groupby restart per group.

Check yourself

1What does groupby('store_id')['revenue'].transform('sum') return?
2How do you compute each line's share of its store's revenue?
3What does groupby('customer_id').filter(lambda g: len(g) >= 5) keep?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.