Module 8: Analyses employers ask for · Lesson 8.6 · about 25 min
Analysing an A/B test in pandas
Read a real-shaped A/B test file, check the split, compute conversion and revenue per visitor for each group, the lift with a 95 % confidence interval and a p-value using only pandas and NumPy, and look at segments without fooling yourself.
Module 8 · about 25 minutes · uses Statistics Module 7. You’ll be able to go from a raw experiment file to a readout, in code.
In everyday terms: a fair taste test: first check both groups really had the same number of tasters, then count who preferred each cup, then ask whether the gap could be chance.
This lesson uses the experiment file from the Statistics course: 24,000 visitors split between the old checkout (A) and a new one (B). The ideas behind each number are in Statistics Lesson 7.5; here we do it in pandas.
1. Look and check the split
A p-value well above 0.01 means the split looks healthy. Each row is one visitor: converted is 1 if they bought.
2. The headline numbers
These match the Statistics capstone: about +0.75 points, CI from about +0.23 to +1.28, p ≈ 0.005. Everything came from one groupby; the rest is arithmetic on two rows.
3. Guardrails and segments
- Guardrails: B is about 46 ms slower to load. Is that acceptable? A decision for the team, written in the readout.
- Segments: B wins on both phones and computers, which makes the overall result more believable. If one segment flipped, treat it as a question for a new test, not a finding.
- Sample ratio check
- Users per group should match the planned split (Statistics Lesson 7.5).
- groupby('group').agg(users=..., conv=...)
- One row per group.
- rate = conv ÷ users
- Conversion per group.
- Lift
- Difference in rates, in percentage points, with a confidence interval.
- Physical meaning
- The lift is a difference between two proportions, each with sampling noise. Its standard error shrinks with more users, so the same 0.5-point lift can be convincing with 50,000 users and pure noise with 2,000.
“B won, it has the higher rate.” Every test has a winner by chance. Report the lift with its interval and p-value, and check the split and guardrails first.
Watch it happen, one step at a time:
| user | group | converted |
|---|---|---|
| 1 | A | 0 |
| 2 | B | 1 |
| 3 | A | 1 |
| ... | ... | ... |
value_counts() → A 5,012, B 4,988: close to 50/50.groupby('group')['converted'].agg(['size', 'sum', 'mean'])| group | users | conv | rate |
|---|---|---|---|
| A | 5012 | 501 | 10.0 % |
| B | 4988 | 549 | 11.0 % |
4. Your turn
How it works
Principle: an A/B readout is a comparison of two means with an honest error bar. For conversion:
The p-value uses the pooled rate, because under “no difference” both groups share one rate. erf turns a z-score into a normal tail area, so no extra library is needed.
Revenue per visitor is very skewed (most visitors spend 0). The mean is still the right number to compare, but use a bootstrap or Welch’s t-test for its interval (Statistics Lessons 4.5 and 6.3).
5. Practice
- Conversion by day of the test: is B ahead every day?
- Why compare revenue per visitor, not revenue per buyer?
- Add a 95 % CI for the conversion of group A alone.
Answers
ab.pivot_table(index='day', columns='group', values='converted', aggfunc='mean'); expect it to wobble day to day.- Per buyer ignores how many people bought; B could make fewer, bigger orders and look better while earning less.
p ± 1.96 * sqrt(p * (1 - p) / n)with A’spandn.
Try it on your own data: bayesian A/B test and sample size check a result or plan the next test.
New words: readout, sample ratio, lift, pooled rate, guardrail metric, segment.
- Check the sample ratio before any result.
- One groupby gives users, conversions and rate per group.
- Report the lift in points with a confidence interval.
- Check guardrail metrics and pre-planned segments only.
Check yourself
Sources
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments, Cambridge University Press
- McKinney, W. (2022), Python for Data Analysis, 3rd edition, O'Reilly (open access) https://wesmckinney.com/book/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.