All lessons

Module 7: A/B testing · Lesson 7.1 · about 25 min

Designing an A/B test

The parts of a good online experiment: a clear hypothesis, the unit you randomise, a fair random split, one primary metric, a planned sample size and duration, and a written decision rule, all fixed before the test starts.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 25 minutes · no new maths. You’ll be able to design an A/B test: state a hypothesis, choose the randomisation unit and metric, and write a plan that holds up when the results come in.

A good A/B test is planned before it starts: one hypothesis, a fair random split, one deciding metric, a size and a duration, and a decision rule written down.

In everyday terms: a fair taste test. You decide in advance which two recipes, who tastes (picked at random, not friends), what counts as “better”, how many tasters, and what you will do with the answer, before anyone takes a bite.

1. Why experiments

Observational data (Lesson 4.2) is full of self-selection: customers who use a feature differ from those who don’t. An A/B test (randomised controlled experiment) removes that problem by letting chance decide who gets the change.

How an A/B test works Visitorseligible for the test Randomsplit 50/50 A: currentcontrol B: new versiontreatment Comparedifference + interval→ decision same metric, same period fibub.com
Randomisation is what makes the comparison fair: anything else that affects the metric (device, season, how keen the customer is) ends up spread evenly between A and B, so the only systematic difference is the change you made.

2. The parts of a design

1. Hypothesis. “Showing delivery dates on product pages will raise purchase conversion, because uncertainty about delivery stops people buying.” A because gives you something to learn even if the test loses.

2. Randomisation unit. Who gets assigned: user, account, session, store, region. Choose the unit that experiences the change and that you analyse. Usually the user: assigned by a stable ID so they see the same version on every visit. If you randomise users, analyse per user (not per page view), or the standard errors are wrong.

3. Population and trigger. Who is eligible (all visitors? logged-in? Canada only?), and when does someone count as “in” the test? Only include people who could see the difference, such as visitors who reached checkout for a checkout change. Including others adds noise.

4. Split. Usually 50/50 for the most power. A smaller treatment share (10 %) limits risk but needs a longer test.

5. Metrics. One primary metric, a few secondary, and guardrails (next lesson).

6. Sample size and duration. From the baseline, the smallest worthwhile effect and power (Lesson 7.3). Run whole weeks (weekday and weekend behaviour differ) and at least one full business cycle.

7. Decision rule. Written before launch: “Ship if the 95 % CI for the conversion lift is above 0 and no guardrail worsens significantly; otherwise keep the current page.”

3. Before launch

  • A/A test (both groups get the same version) to check the pipeline: you should see no difference and a correct split.
  • Check the assignment is random and stable: the split should match the plan (Lesson 7.6).
  • Freeze the plan in a document with a date.
Hypothesis
What will change, for whom, and why.
Unit of randomisation
What gets assigned to A or B: a user, a session, a store. Analyse at the same level.
Primary metric
The one number that decides.
Decision rule
Written before launch: "ship if the 95 % CI for the lift is above 0 and no guardrail is breached".
Physical meaning
Randomisation is what lets you say "caused". Because chance alone decides who sees B, the two groups are alike in everything except the change, so a difference in results can only come from the change or from luck, and statistics measures the luck.

“We can decide what success means after we see the data.” Choosing the metric, segment or end date after looking lets luck pick your winner. Write the plan first and keep extra findings as ideas for the next test.

Watch it happen, one step at a time:

Hypothesis. Showing delivery dates raises purchase conversion because it removes uncertainty.
Unit. Logged-in users, split 50/50 by a hash of their ID, so each person always sees the same version.
Metric. Primary: share of users who buy. Guardrails: page load time, returns.
Size. Enough users to detect +5 % relative with 80 % power: work it out (Lesson 7.3), round up to whole weeks.
Rule. Ship if the 95 % CI for the lift is above 0 and no guardrail is breached. Written down before launch.
Maths and principle

Principle: with random assignment, the expected difference between groups in anything other than the treatment is zero, so the measured difference estimates the treatment’s causal effect.

Equation: average treatment effect

ATE=E[Y|B]−E[Y|A]

estimated by y‾B−y‾A with standard error sA2/nA+sB2/nB (Lesson 4.4).

Hashing: a common way to assign users stably is hash(user_id + test_name) mod 100 < 50 → B. Including the test name keeps different tests’ assignments independent.

4. Worked example

Made-up plan for “show delivery dates”:

Item Plan
Hypothesis Delivery dates raise purchase conversion by reducing uncertainty
Unit Logged-in user, hashed ID, 50/50
Trigger User views a product page during the test
Primary metric Share of triggered users who purchase within 7 days
Secondary Add-to-cart rate; average order value
Guardrails Cancellations; “where is my order” contacts; page load time
Baseline / MDE 4.0 %; smallest worthwhile lift +10 % relative (to 4.4 %)
Size / duration About 39,500 users per group; at 6,000 eligible users a day, 14 days (2 full weeks)
Decision rule Ship if 95 % CI for lift > 0 and no guardrail worse at p < 0.05

5. Practice

  1. A store tests a new shelf layout. What is a sensible randomisation unit, and why not the customer?
  2. Why should a checkout test only include users who reach checkout?
  3. Write a decision rule for a test whose goal is “don’t hurt revenue while simplifying the menu”.
Answers
  1. The store (or store-week): customers can’t be shown different shelves in the same store. With store-level randomisation, analyse per store, and you’ll need many stores.
  2. Users who never reach checkout can’t be affected; including them adds noise and dilutes the effect, so you need more traffic for the same power.
  3. Example: “Ship the simpler menu if the 95 % CI for the change in revenue per user is above −1 % (a non-inferiority test) and support contacts don’t rise.”

Try it on your own data: the sample size calculator plans the test size; the bayesian A/B test tool reads finished tests.

New words: A/B test, randomised controlled experiment, randomisation unit, trigger, A/A test, decision rule, average treatment effect, non-inferiority.

  • Write the hypothesis, unit, metrics, size, duration and decision rule before launch.
  • Randomise the same unit you analyse (usually the user).
  • Run whole weeks to cover weekday patterns.
  • Randomisation is what turns a difference into evidence of cause.

Check yourself

1Why randomise who sees the new version instead of letting users opt in?
2A pricing test randomises page views, so one customer can see both prices. What is the problem?
3What should be written down before an A/B test starts?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 2-3
  • OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 1.4 Experiments https://www.openintro.org/book/os/

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.