All lessons

Module 7: A/B testing · Lesson 7.2 · about 20 min

Choosing metrics: primary, secondary and guardrails

One metric decides; a few explain; guardrails protect. How to pick a primary metric that is sensitive and tied to value, ratio metrics and the unit trap, and guardrails that stop a win from hiding a loss.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 20 minutes · no new maths. You’ll be able to choose a primary metric, secondary metrics and guardrails, and avoid the unit mismatch in ratio metrics.

One metric decides, a few explain, and guardrails make sure the win does not break something else.

In everyday terms: a diet plan judged on weight lost (the deciding metric), with energy and sleep tracked to explain how (secondary), and blood pressure watched so you do not win the diet and lose your health (guardrail).

1. One decides, some explain, some protect

Metrics for a checkout test Goal: more completed orders Primary: purchase conversiondecides ship / don't ship Secondary (explain why)checkout starts · payment errorsaverage order value · time to pay Guardrails (must not worsen)refunds · support contactspage load time · revenue per visitor fibub.com
Write this tree before the test starts. Only the primary metric decides the outcome; secondary metrics help explain it; a guardrail that gets clearly worse can stop a launch even if the primary metric wins.

Primary metric (decides): the one number the decision rule uses. A good one is

  • tied to value (purchases, revenue per user, retained users), not just activity;
  • sensitive enough to move within the test’s length (annual retention won’t);
  • hard to game (clicks are easy to raise by making things confusing).

Secondary metrics (explain): the steps that should move if the hypothesis is right: add-to-cart, checkout starts, average order value. They tell you why the primary moved or didn’t.

Guardrails (protect): things that must not get worse: refunds, complaints, page speed, unsubscribe rate, revenue. A win on the primary with a clear loss on a guardrail is not a win.

2. Common metric types

Type Example Analysed with
Proportion per user Did the user buy? Two-proportion z-test (Lesson 7.5)
Mean per user Revenue per user, orders per user Welch t-test or bootstrap
Ratio of totals Revenue ÷ orders (AOV), clicks ÷ views (CTR) Delta method or bootstrap by user
Time to event Days until second order Survival methods (beyond this course)

The ratio trap. If you randomise users but compute “clicks ÷ page views” and test it as if each page view were independent, the SE is too small and false positives soar. Either turn it into a per-user metric (each user’s CTR, then average) or keep the ratio and resample users.

Revenue per user is often the best primary for commerce, but it is very noisy (mostly zeros, a few big orders). Options: a longer test, capping extreme values at, say, the 99th percentile (decided in advance), or CUPED (adjusting for each user’s spend before the test), which can cut variance a lot.

3. Overall evaluation criterion

Some teams combine several metrics into one overall evaluation criterion (OEC) with agreed weights, such as revenue minus a cost per support contact. It makes trade-offs explicit before the test, not after.

Primary metric
Sensitive to the change and tied to value; decides the test.
Secondary metrics
Explain the result: clicks, items per order.
Guardrails
Must not get worse beyond a set limit: load time, refunds, complaints.
Ratio metrics
Per user, not per page view, when the user is the randomised unit; otherwise heavy users dominate and SEs come out wrong.
Physical meaning
A metric is the question you ask the experiment. Ask about clicks and you get click-bait; ask about revenue per user and you learn whether customers are really better off.

“Pick the metric that moved most.” With ten metrics, one will move by luck. Name the primary metric in the plan and treat the others as explanation or warning lights.

Watch it happen, one step at a time:

The test. A "Recommended for you" row on the home page.
Primary. Revenue per user over 14 days, capped at the 99th percentile (decided in advance): +1.8 % (CI +0.2 % to +3.4 %).
Secondary. Clicks on the row, items per order +3 %: the extra revenue comes from bigger baskets.
Guardrail. Load time +140 ms, beyond the 100 ms limit.
Decide. Do not ship yet: make the row load lazily, then re-test. The win is real but the guardrail is breached.
Maths and principle

Principle: the analysis must respect the randomisation unit, because only those units are independent.

Delta method for a ratio R=Y‾/X‾ (per-user sums Yi, Xi):

Var(R)≈1nX‾2(sY2−2RsXY+R2sX2)

Variance reduction (CUPED): with each user’s pre-test value Xi, analyse Yi−θ(Xi−X‾), θ=cov(X,Y)/var(X). Variance falls by the factor 1−r2, where r is the correlation between before and during.

4. Worked example

Made-up test: a “Recommended for you” row on the home page.

  • Primary: revenue per user over 14 days (capped at the 99th percentile, decided in advance).
  • Secondary: click-through on the new row; items per order; share of users who buy.
  • Guardrails: home page load time (must not rise > 100 ms); returns rate.
  • Result: revenue per user +1.8 % (95 % CI +0.2 % to +3.4 %); buyers +0.4 points; items per order +3 %; load time +140 ms (CI +120 to +160).
  • Decision: the primary wins but a guardrail clearly fails. Fix the load time (lazy-load the row) and retest, rather than ship.

5. Practice

  1. Pick a primary metric for a test of a simpler sign-up form.
  2. Name two guardrails for a test of more aggressive push notifications.
  3. Pre-test and in-test spend correlate at r = 0.6. By how much does CUPED cut the variance?
Answers
  1. Share of visitors who complete sign-up (and, if possible, who are still active a week later, so a form that signs up the wrong people doesn’t win).
  2. Notification opt-outs and app uninstalls; complaints; long-run active users.
  3. By the factor 1 − 0.36 = 0.64, so 36 % less variance, equivalent to about 56 % more traffic.

Try it on your own data: the bootstrap test tool resamples by row, so put one row per user to get honest intervals for per-user metrics.

New words: primary metric, secondary metric, guardrail metric, overall evaluation criterion, ratio metric, delta method, CUPED, capping.

  • One primary metric, chosen before the test.
  • Secondary metrics explain; guardrails protect.
  • Use per-user metrics when users are randomised.
  • Cap or transform extreme values in revenue metrics, as planned in advance.

Check yourself

1Why choose a single primary metric?
2A test randomises users but reports "clicks per page view". What is the risk?
3Which is a guardrail for a test of bigger "Buy now" buttons?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 6-7 (metrics)
  • Deng, A. and Shi, X. (2016), Data-Driven Metric Development for Online Controlled Experiments, KDD 2016 https://doi.org/10.1145/2939672.2939700

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.