Cohen's Kappa: Measuring Agreement Beyond Chance

Two doctors independently read the same 100 scans and label each "abnormal" or "normal." They agree on 75 of them. Is 75% agreement good? It depends on how much agreement two people would reach by luck alone. If nearly every scan is normal, two raters who guessed "normal" almost every time would agree very often without any skill at all. Cohen's kappa corrects raw agreement for the agreement expected by chance, and it is the standard way to report how consistently two raters, coders, or classifiers label the same items into categories.

The Formula

Kappa compares the agreement actually observed, pₒ, with the agreement expected by chance, pₑ:

The numerator is how much better the raters did than chance; the denominator is the most they could possibly have improved on chance. A kappa of 1 means perfect agreement, 0 means agreement no better than chance, and negative values mean agreement worse than chance, which signals systematic disagreement.

Chance agreement is computed from each rater's overall tendencies, using the same independence logic as the expected counts in the chi-square test: if the two raters labelled items independently, each category would be chosen by both with probability equal to the product of their individual rates.

Worked Example

Problem: Two radiologists classify 100 scans as abnormal (Yes) or normal (No):

	Rater 2: Yes	Rater 2: No	Total
Rater 1: Yes	45	15	60
Rater 1: No	10	30	40
Total	55	45	100

Step 1 — Observed agreement. The raters agree on the diagonal cells: 45 + 30 = 75 scans, so pₒ = 75 ÷ 100 = 0.75.

Step 2 — Chance agreement. Rater 1 says Yes 60% of the time and Rater 2 says Yes 55% of the time. By chance, both say Yes with probability 0.60 × 0.55 = 0.33, and both say No with probability 0.40 × 0.45 = 0.18. So pₑ = 0.33 + 0.18 = 0.51. More than half the scans would be agreed on by luck alone.

Step 3 — Kappa.

A 75% raw agreement shrinks to a kappa of about 0.49 once chance is removed. The raters recovered roughly half of the achievable improvement over guessing.

Uncertainty in Kappa

Like any statistic from a sample, kappa has sampling error. A simple approximate standard error is √[pₒ(1 − pₒ) ÷ (n(1 − pₑ)²)] = √[0.75 × 0.25 ÷ (100 × 0.2401)] ≈ 0.088. An approximate 95% confidence interval is 0.49 ± 1.96 × 0.088, or about 0.32 to 0.66. This is a wide range, spanning what many would call "fair" to "substantial" agreement, so with only 100 items the data support only a loose conclusion. More refined standard errors exist, and software typically reports one; the approximation here is meant to convey the general size of the uncertainty, as discussed in confidence intervals.

When High Agreement Misleads

Kappa earns its keep when categories are unbalanced. Suppose the two raters now label 100 scans where abnormalities are rare:

	Rater 2: Yes	Rater 2: No	Total
Rater 1: Yes	2	3	5
Rater 1: No	4	91	95
Total	6	94	100

Raw agreement is (2 + 91) ÷ 100 = 93%, which sounds excellent. But the raters say Yes only 5% and 6% of the time, so chance agreement is 0.05 × 0.06 + 0.95 × 0.94 = 0.003 + 0.893 = 0.896. Then κ = (0.93 − 0.896) ÷ (1 − 0.896) = 0.034 ÷ 0.104 ≈ 0.33. The raters agree on 91 normal scans, which nearly anyone would, but of the 9 scans that either flagged as abnormal, they agreed on only 2. The headline 93% mostly reflects how common normal scans are, not the raters' skill at the hard part, the same trap as quoting plain accuracy in sensitivity and specificity.

Interpreting the Value

A widely quoted guide from Landis and Koch labels 0.01 to 0.20 as slight, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and above 0.80 as almost perfect. On that scale, the first example (0.49) is moderate and the second (0.33) is fair. These labels are conventions, not laws. What counts as acceptable depends on the stakes: agreement of 0.6 might be fine for coding survey comments and inadequate for diagnosing a disease. Kappa is also limited by the raters' marginal rates. In the first example, given each rater's Yes rate of 60% and 55%, the best kappa achievable is about 0.90, not 1.00, because perfect agreement would require identical marginals.

Extensions

• More than two categories: the same formula works with a larger square table; observed agreement sums the whole diagonal, and chance agreement sums the products of matching row and column proportions.
• Ordered categories: weighted kappa gives partial credit for near misses, so disagreeing between "mild" and "moderate" costs less than between "mild" and "severe." It suits ratings like the Likert scales discussed in types of data.
• More than two raters: Fleiss' kappa generalizes the idea to any fixed number of raters.

Real-World Uses

Medical imaging and diagnosis: reporting how consistently clinicians classify the same cases, a prerequisite for trusting a diagnostic category.

Content analysis and qualitative research: when two coders label interview passages into themes, kappa shows whether the coding scheme is clear enough to be applied consistently.

Machine learning: comparing a model's labels with human annotations, or checking agreement between annotators who create training data, as part of the workflow described in statistics in machine learning.

Common Mistakes

Quoting percent agreement alone. It ignores chance and can look impressive for reasons that have nothing to do with skill.

Applying kappa to paired measurements of the same quantity. Kappa is for categorical labels. For continuous measurements, other tools such as the correlation and the intraclass correlation are appropriate.

Comparing kappas across datasets with different category rates. The same raters can produce different kappas simply because the prevalence of a category differs.

Treating labels as benchmarks. "Moderate" and "substantial" are rules of thumb. State the kappa and its interval, and let readers judge against the stakes.

What is a good kappa value?

Common guidelines call 0.61 to 0.80 substantial and above 0.80 almost perfect, but acceptable levels depend on the field and the consequences of disagreement. High-stakes decisions demand higher agreement.

Can kappa be negative?

Yes. A negative value means the raters agree less often than chance would produce, which usually points to a systematic problem such as reversed coding of categories.

Is kappa the same as accuracy?

No. Accuracy, or percent agreement, is the raw proportion of matching labels. Kappa adjusts that proportion for the level of agreement expected by chance alone, so it is always lower than or equal to the raw agreement when agreement is above chance.

How is kappa different from McNemar's test?

Kappa measures how strongly two raters agree. McNemar's test asks whether the two raters differ in how often they say Yes. Two raters can agree strongly and still have a systematic difference in tendency, or disagree while having equal rates.