Module 7: A/B testing · Lesson 7.6 · about 15 min
Sample ratio mismatch
When a 50/50 test ends with 50.5 % of users in one group, something in the setup is broken and the result cannot be trusted. How to test the split with a chi-square, and the usual causes.
Module 7 · about 15 minutes · uses Lesson 6.5. You’ll be able to test whether a split matches the plan and list the common causes when it doesn’t.
In everyday terms: you deal a deck of cards to two players and one ends up with 30 cards. You do not argue about who won the hand; you check how the cards were dealt.
1. Why the split matters
Randomisation makes the groups alike in everything except the change (Lesson 7.1). If users go missing from one group, or extra ones arrive, the groups stop being alike, and the result measures who is in each group as well as the change. That is a sample ratio mismatch (SRM).
It’s surprisingly common. Large experimentation teams report it in a few percent of all tests, and it usually reveals a real bug.
Move the visitors slider: the band of “normal” shares narrows as the test grows. A 50.5 % share is nothing with 1,000 users and a red flag with 200,000.
2. The check
Compare the observed counts with the planned split using a chi-square goodness-of-fit test (Lesson 6.5) with one degree of freedom. Run it on every test, before reading any metric, and use p < 0.001 as the alarm.
3. Common causes
- Redirects or extra page loads in one variant: users drop off (or get counted twice) before logging.
- Tracking that fails more in one variant: a heavier page, a script that loads late, an ad blocker that blocks one design.
- Bot filtering that treats one variant differently.
- Assignment bugs: the hash isn’t stable, users switch groups, or a test started on one platform earlier than the other.
- Triggering on something the change affects: “users who clicked the new button” exists only in B.
- A ramp (10 % → 50 %) analysed as if it were one 50/50 test.
What to do: don’t “fix” it by dropping users. Find the cause, fix it, and rerun. Sometimes a mismatch limited to one browser or one day can be cut out and the rest analysed, but say so plainly.
- Expected counts
- Total × planned share for each group.
- χ² = Σ (O − E)² ÷ E
- How far the actual split is from the plan; 1 df for two groups.
- SRM
- Sample ratio mismatch: a split so uneven that chance is not a believable explanation (p below about 0.001).
- Physical meaning
- A random split cannot go far wrong on large numbers. When it does, some users are being lost or added in one group: a crash, a redirect, a bot filter. Those missing users are rarely typical, so they bias the comparison.
“The split is only off by 1.5 %, so it is close enough.” With 100,000 users, a 51.5/48.5 split is wildly unlikely by chance (p far below 0.0001). Small percentage gaps on big samples are red flags.
Watch it happen, one step at a time:
Maths and principle
Principle: the split itself is a random outcome with a known distribution, so it can be tested like any other.
Equation (observed ; expected , for planned shares ):
Check: 101,000 vs 99,000 against 100,000 each: , p ≈ 0.000008. The same 50.5 % share with 1,000 users (505 vs 495) gives , p = 0.75.
Rule of thumb: for a 50/50 split, B’s share has SD ; the 99.9 % band is about .
4. Worked example
Made-up story. A test of a new product page shows +4 % revenue per user. The split: A 52,140, B 50,610 (planned 50/50).
- Expected 51,375 each. , p ≈ 0.000002. SRM.
- Breaking it down by browser: the gap is all in one older mobile browser, where B’s new image gallery crashed the page before the tracking fired.
- Those users (likely on older, cheaper phones, who spend less) are missing from B, which explains much of B’s “higher revenue per user”.
- Decision: no conclusion. Fix the gallery, rerun.
5. Practice
- 50/50 test: 5,060 vs 4,940. SRM?
- 90/10 test (90 % control): 90,500 vs 9,500. SRM?
- Why not just reweight the groups to 50/50?
Answers
- Expected 5,000 each: , p = 0.23. Fine.
- Expected 90,000 and 10,000: , p < 0.000001. SRM.
- Reweighting fixes the counts, not the mix: the missing users are a particular kind of user, and weighting the rest can’t bring them back.
Try it on your own data: the cross-tab and chi-square tool runs goodness-of-fit tests on counts; the readout in Lesson 7.5 checks the split automatically.
New words: sample ratio mismatch (SRM), goodness of fit, ramp-up, triggering bias.
- Always test the split before reading results.
- χ² on the group counts against the planned shares; p < 0.001 means SRM.
- Look for the cause by browser, device, day and entry point.
- An SRM test gives no trustworthy result; fix the cause and rerun.
Check yourself
Sources
- Fabijan, A. et al. (2019), Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD 2019 https://doi.org/10.1145/3292500.3330722
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapter 21
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.