Module 7: A/B testing · Lesson 7.7 · about 25 min
Bayesian A/B testing
Instead of a p-value, get the chance that B is better and the expected loss if you pick it. How Beta posteriors work for conversion rates, what the numbers mean, and where Bayesian testing does and does not save you.
Module 7 · about 25 minutes · uses Lessons 2.2 and 7.5. You’ll be able to read a Bayesian A/B result, explain “chance B is better” and “expected loss”, and say when the approach helps.
In everyday terms: a weather forecast says “70 % chance of rain” instead of “we reject the hypothesis of a dry day”. You decide about the umbrella from the chance and from how much you mind getting wet.
1. The question managers actually ask
A p-value answers: “if there were no difference, how surprising is this?” Managers usually want: “what’s the chance B is better, and how much do we risk if we pick it?” Bayes’ rule (Lesson 2.2) answers that directly, at the cost of stating a prior: what you believed about the rates before the test.
Start with 80 vs 100 conversions out of 2,000 each. The curves overlap a little: B is probably better, but not certainly. Raise the visitors (keeping the rates the same) and watch the curves narrow and separate.
2. Posteriors for conversion rates
For a conversion rate, a convenient prior is a Beta distribution. With a flat prior Beta(1, 1), after conversions from visitors the posterior is Beta(, ): a curve of how plausible each rate is.
From the two posteriors you can read:
- P(B better than A): the share of plausible (A, B) pairs where B is higher.
- Expected loss of choosing B: the average shortfall if B is actually worse, counting the cases where B is better as zero loss.
- Credible interval for B − A: the middle 95 % of plausible differences.
Teams typically agree in advance to ship when the expected loss is below a tiny threshold (say 0.01 percentage points), which they can turn into money.
3. Priors
- A flat prior lets the data speak; with thousands of visitors its effect is negligible.
- An informative prior from past tests (most changes move conversion by less than a few percent) shrinks wild early results towards zero. That’s a strength, but it must be set before the test and written down.
- Prior
- What you believed about the rate before the test (often a flat Beta(1, 1)).
- Posterior Beta(1 + buyers, 1 + non-buyers)
- What you believe after the data, for each version.
- P(B > A)
- The share of posterior draws where B's rate beats A's.
- Expected loss
- If you pick B, the average amount by which A would have been better, counting only the cases where it is.
- Physical meaning
- The posterior is a cloud of plausible true rates. Draw from both clouds many times: how often B lands above A is the chance B is better, and how far below A it lands when it loses is the risk.
“P(B better) = 94 % means ship.” It depends on what being wrong costs. Decide in advance on an expected-loss threshold; a 94 % chance with a tiny downside can be a fine bet, and with a big downside it is not.
Watch it happen, one step at a time:
Maths and principle
Principle: posterior ∝ likelihood × prior; for 0/1 outcomes the Beta prior stays Beta after the data (it is conjugate).
Update:
Chance B is better and expected loss (estimated by drawing many pairs from the two posteriors):
Check with the starting numbers: Beta(81, 1921) and Beta(101, 1901) give P(B better) ≈ 93.6 %, expected loss ≈ 0.018 points, 95 % credible interval −0.29 to +2.30 points (a million draws in Python). The classic two-sided p-value is 0.13; half of it, 0.064, is close to 1 − 0.936, as it usually is with a flat prior.
4. Worked example
Made-up test: new product photos. A: 80 of 2,000 bought (4.0 %). B: 100 of 2,000 (5.0 %).
- Frequentist: +1.0 point, 95 % CI −0.3 to +2.3, p = 0.13. “Not significant.”
- Bayesian: 93.6 % chance B is better; if you pick B and it’s actually worse, you lose on average 0.018 points.
- The decision rule agreed beforehand: ship when expected loss < 0.01 points. Not yet met; keep running to the planned size.
- Two weeks later, at 4,000 per group with the same rates, P(B better) is about 98 % and expected loss is about 0.003 points, under the 0.01 threshold. Ship.
Both views agree on the evidence. The Bayesian one states it in terms people can act on.
5. Where it doesn’t help
- Peeking: in a quick simulation (no real difference, 4 % conversion, 500 users a group a day, checked daily for 28 days), stopping the first time P(B better) > 95 % gave a false win about 24 % of the time, close to the frequentist figure in Lesson 7.4.
- Bad data: SRM, bots and broken tracking break any method.
- Many metrics: checking ten metrics’ “chance of being better” still invites cherry-picking.
6. Practice
- With a flat prior, what is the posterior after 30 conversions out of 600?
- P(B better) = 80 %, expected loss 0.002 points, threshold 0.01. Ship?
- Why might a team prefer an informative prior centred on zero lift?
Answers
- Beta(31, 571).
- Yes by that rule: B is not certainly better, but the likely downside is tiny.
- Most changes do little, so a prior centred on zero tames lucky early swings and reduces false wins.
Try it on your own data: the bayesian A/B test tool computes posteriors, P(B better) and expected loss for your counts; probability distribution explorer draws Beta curves.
New words: prior, posterior, Beta distribution, conjugate prior, credible interval, expected loss.
- With a flat prior, the posterior for a rate is Beta(1 + yes, 1 + no).
- P(B > A) and expected loss come from comparing draws from the two posteriors.
- Set the decision threshold (expected loss) before the test.
- Bayesian tests are not immune to peeking problems unless the rule is planned.
Check yourself
Sources
- Stucchio, C. (2015), Bayesian A/B Testing at VWO, VWO technical paper https://vwo.com/downloads/VWO_SmartStats_technical_whitepaper.pdf
- Gelman, A. et al. (2013), Bayesian Data Analysis, 3rd edition, CRC Press, Chapter 2 (the beta-binomial model) http://www.stat.columbia.edu/~gelman/book/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.