Tooth wear scoring on intraoral scans: what weighted kappa shows about BEWE and TWES 2.0
Two examiners scored tooth wear on 246 intraoral scans with good agreement. We show with fibub's reliability tool how the weighting choice, rater bias and sample size move kappa.
Dentists grade tooth wear by eye, and two dentists do not always give the same grade. If scoring is moved from the mouth to a 3D intraoral scan, the scan can be stored and scored again later, but only if the grades are reliable. A new study had two examiners grade 246 scans with two common systems, BEWE and TWES 2.0, and measured their agreement with weighted kappa. This article shows how the reliability and agreement tool works out kappa from an agreement table, and how three choices that papers rarely spell out move the answer. The sample size calculator helps plan the number of subjects, and the bland-Altman agreement plot tool is the right one when the two measures are numbers rather than grades.
The paper in brief
The Basic Erosive Wear Examination (BEWE) gives each sixth of the mouth (a sextant) a score from 0 (no wear) to 3 (wear over half the surface). TWES 2.0 is a newer system with grades 0 to 4. Two experienced examiners scored 246 adult scans independently.
For BEWE, weighted kappa ran from 0.760 to 0.851 across sextants, 0.841 overall. For TWES 2.0 it was lower, 0.543 to 0.761, 0.715 overall. On the usual scales, BEWE agreement is good to very good, TWES 2.0 moderate to good. One examiner tended to score higher than the other. The authors planned the study to show agreement clearly above a kappa of 0.40, which needed 49 to 75 sextants for BEWE; they had about 240 per sextant.
Checked with the tools
The paper’s agreement tables are images we did not reuse, and it does not say which weights it used. So we built a simulated agreement table for one BEWE sextant with 246 scans (our own numbers, shaped like a typical result: most sextants score 0 or 1, rater 1 a little higher), and ran it through the Reliability analysis tool’s maths.
Step 1: what kappa measures. The two simulated raters give the same grade on 81.3% of sextants. But with most sextants at 0 or 1, they would agree on 32.0% by chance alone. Kappa is the share of the remaining room they actually use: (0.813 − 0.320) ÷ (1 − 0.320) = 0.725, 95% interval 0.653 to 0.797.
Step 2: the weighting choice. BEWE grades are ordered, so a 1 against a 2 is a smaller miss than a 0 against a 3. Weighted kappa gives partial credit for near misses. On the same table, linear weights give 0.791 and quadratic weights 0.860. The paper’s BEWE range, 0.760 to 0.851, sits between them. Without the weighting stated, a reader cannot tell which part of that spread is the raters and which is the formula. Quadratic weights give the most credit for near misses and usually the highest value.
Step 3: a rater who scores higher. We moved a growing share of the agreed grades one step up for rater 1 (Figure 1). At 10%, unweighted kappa falls from 0.725 to 0.615, linear to 0.715, quadratic to 0.816. A steady bias costs unweighted kappa most. Kappa does not tell you which rater is higher; look at the table above and below the diagonal, as the paper did with a Wilcoxon test. In our table, rater 1 is higher on 29 sextants and lower on 17.
Step 4: how sure is a kappa from 60 sextants? We drew 2,000 simulated studies from the same table (Figure 2). With 60 sextants, unweighted kappa varies with a standard deviation of 0.075; with 246, 0.036. Both sizes almost always show agreement above 0.40 (97% and 100% of studies had their 95% interval above it), in line with the paper’s planning. But to tell 0.76 from 0.85, the bigger sample is needed.
| Check | Paper | Tool result (simulated table) |
|---|---|---|
| BEWE agreement, one sextant | weighted κ 0.760 to 0.851 | 0.725 unweighted, 0.791 linear, 0.860 quadratic |
| Effect of one rater scoring higher | rater 1 generally higher | 10% upward shift: κ 0.725 → 0.615 |
| Precision with 60 vs 246 sextants | planned N 49 to 75 | SD of κ 0.075 vs 0.036 |
| Agreement above κ = 0.40 | yes, all sextants | 97% of 60-sextant studies, all 246-sextant ones |
How we made the graphs
The charts and every number above come from scripts/blog-figures/bewe-twes-tooth-wear-intraoral-scan-weighted-kappa.mjs in fibub’s public code. It calls cohen from src/lib/sci/reliability.ts, the function behind the Two raters mode of the reliability and agreement tool. It works out observed and chance agreement, kappa with no weights or with linear or quadratic weights, and the large-sample standard error for unweighted kappa.
From the paper: 246 scans, the 0 to 3 BEWE scale, the reported kappa range, the fact that one rater scored higher, and the planning value of 0.40. Our assumptions, labelled in the script: the 4 × 4 agreement table, the way the upward bias is added, and the 2,000 random studies drawn from the table.
How we test the maths
On every change to fibub, tests/sci.mjs checks the reliability maths:
- A 2 × 2 table of 20, 5, 10 and 15 must give Cohen’s kappa of exactly 0.4 (observed 70%, chance 50%).
- Perfect agreement on three ordered categories must give quadratic weighted kappa of exactly 1.
- The same file pins Cronbach’s alpha, Fleiss’ kappa and the intraclass correlations against published worked examples.
Check it yourself
On the reliability and agreement tool:
- Press Two raters. Leave My data is on A table of counts.
- In the box under Counts: rows are rater 1’s categories, columns rater 2’s, replace the example with the four rows below (BEWE grades 0 to 3 in order).
- With Weights on None (categories) it reads κ = 0.725, 95 % interval 0.653 to 0.797, Observed agreement 81.3 %, Expected by chance 32.0 %.
- Switch Weights to Linear (ordered): κ = 0.791. Then Quadratic (ordered): κ = 0.860.
70 8 1 0
14 80 6 0
1 10 40 2
0 0 4 10
What researchers can use it for
- Report agreement fully: give the table, the weights used and the observed agreement alongside kappa, all of which the reliability and agreement tool shows at once.
- Compare studies fairly: recompute kappa with the same weights before comparing a new scoring system with an old one.
- Spot a biased rater: read the counts above and below the diagonal, not only the kappa.
- Plan a reliability study: simulate the spread of kappa at a few sample sizes, or use the sample size calculator for simpler designs.
- Use the right tool for numbers: for continuous measures, such as wear depth in micrometres, use bland-Altman agreement plot limits of agreement instead of kappa.
The fine print
- The kappa ranges, sample size and design come from the paper. Our agreement table is simulated and does not reproduce any of the paper’s sextants.
- The paper does not say whether its weights were linear or quadratic, so we cannot say which of our values matches its method.
- The tool gives an interval for unweighted kappa only; for weighted kappa, a bootstrap (as in Figure 2) gives one.
- This is about the statistics of the scoring, not dental advice.
Sources
- Lorens, M. and Tomaszewska, I. (2026), A reliability assessment of the basic erosive wear examination and the tooth wear evaluation system 2.0 utilizing intraoral scan data, PLOS ONE 21(6): e0351330, doi:10.1371/journal.pone.0351330. Licence CC BY 4.0, checked on the journal page, 2026-11-16. Values used here, from the Methods and Results: 246 scans, two examiners; BEWE scores 0 to 3 per sextant, TWES 2.0 grades 0 to 4; weighted kappa for BEWE 0.760 to 0.851 across sextants, 0.841 overall and 0.868 for the total score; for TWES 2.0 0.543 to 0.761, 0.715 overall; weighting scheme not stated; one examiner generally scored higher; sample size planned against kappa0 = 0.40 with 80, 90 and 95% power, BEWE required N = 49 to 75.
- Cohen, J. (1960), Educational and Psychological Measurement 20: 37 (kappa); Cohen, J. (1968), Psychological Bulletin 70: 213 (weighted kappa); Fleiss, J. L., Cohen, J. and Everitt, B. S. (1969), Psychological Bulletin 72: 323 (standard error); Landis, J. R. and Koch, G. G. (1977), Biometrics 33: 159 (bands).
- BEWE: Bartlett, D. W., Ganss, C. and Lussi, A. (2008), Clinical Oral Investigations 12 (Suppl 1): S65 (background only).
If you are an author or publisher and want a credit or a passage changed or removed, tell us and we will act quickly.