Sensitivity, Specificity, and ROC Curves: Measuring How Good a Test Really Is

"This test is 90% accurate" sounds reassuring, but it hides at least two different questions: how often does the test catch people who really have the condition, and how often does it wrongly flag people who don't? Those are separate qualities, measured by sensitivity and specificity, and a test can be strong on one while weak on the other. Once a test produces a score rather than a simple yes or no, the ROC curve shows how those two qualities trade off as the cutoff moves. Together these tools are the standard language for judging any classifier, from medical screening to spam filters to fraud detection.

The Four Outcomes

Whenever a test tries to sort cases into "has the condition" and "does not," every case lands in one of four cells. This layout is the confusion matrix:

	Condition present	Condition absent
Test positive	True positive (TP)	False positive (FP)
Test negative	False negative (FN)	True negative (TN)

Sensitivity is the proportion of people who have the condition that the test correctly flags: TP ÷ (TP + FN). Specificity is the proportion of people without the condition that the test correctly clears: TN ÷ (TN + FP). Both are properties of the test itself, measured within one column of the table.

The mirror-image questions are asked within a row. The positive predictive value (PPV), TP ÷ (TP + FP), answers "given a positive result, how likely is the condition?" The negative predictive value (NPV), TN ÷ (TN + FN), answers the same for a negative result. These depend not only on the test but on how common the condition is.

Worked Example

Problem: A screening test is used on 1,000 people, of whom 10% (100 people) truly have the condition. The test has a sensitivity of 90% and a specificity of 80%. Find the full confusion matrix, then the PPV and NPV.

Condition present (100 people): 90% are caught, so TP = 90 and FN = 10.

Condition absent (900 people): 80% are correctly cleared, so TN = 720 and FP = 180.

	Condition present	Condition absent	Total
Test positive	90	180	270
Test negative	10	720	730
Total	100	900	1,000

Only one in three positive results is a true case, even though the test catches 90% of real cases. The reason is the 180 false positives: a 20% false-alarm rate applied to the large healthy group produces twice as many false alarms as the test's true detections. A negative result, on the other hand, is very reassuring, since 98.6% of negatives are truly negative.

Why Prevalence Changes Everything

Sensitivity and specificity stay fixed for a given test, but predictive values do not. Keep the same test and drop the prevalence to 1%: of 1,000 people, 10 have the condition and 990 do not. The test catches 9 of the 10, and falsely flags 20% of the 990, which is 198 people. The PPV falls to 9 ÷ (9 + 198) ≈ 0.043, so fewer than 5% of positives are real. This is the same effect that appears in Bayes' theorem and its medical testing example, seen here from the confusion matrix side: the rarer the condition, the more a positive result is dominated by false alarms.

Likelihood Ratios: A Shortcut Through the Odds

The positive likelihood ratio, sensitivity ÷ (1 − specificity), tells how much a positive result multiplies the odds of the condition. Here it is 0.90 ÷ 0.20 = 4.5. Prior odds were 100 to 900, about 0.111. Multiplying gives posterior odds of 0.111 × 4.5 = 0.5, which converts to a probability of 0.5 ÷ 1.5 = 0.333, exactly matching the PPV found above. Working in odds makes the update a single multiplication.

Thresholds and the ROC Curve

Many tests do not give a yes or no. They give a score, such as a blood marker level or a model's predicted probability from logistic regression, and a threshold decides what counts as positive. Lowering the threshold catches more true cases (higher sensitivity) but also flags more healthy people (lower specificity), and raising it does the opposite. The ROC curve plots this trade-off across every possible threshold, with the false positive rate (1 − specificity) on the horizontal axis and the true positive rate (sensitivity) on the vertical axis.

Example: Ten people are scored by a test. The five who truly have the condition score 0.90, 0.80, 0.60, 0.55, 0.30; the five who do not score 0.70, 0.50, 0.40, 0.35, 0.20. Treating "score at or above the threshold" as positive gives:

Threshold	True positives	False positives	Sensitivity (TPR)	1 − Specificity (FPR)
0.90	1	0	0.2	0.0
0.80	2	0	0.4	0.0
0.70	2	1	0.4	0.2
0.60	3	1	0.6	0.2
0.55	4	1	0.8	0.2
0.50	4	2	0.8	0.4
0.40	4	3	0.8	0.6
0.35	4	4	0.8	0.8
0.30	5	4	1.0	0.8
0.20	5	5	1.0	1.0

Plotting these (FPR, TPR) pairs and joining them traces the ROC curve. A curve that hugs the top-left corner is a strong test; a diagonal line from (0,0) to (1,1) is no better than guessing.

AUC: One Number for the Whole Curve

The area under the curve (AUC) summarizes the ROC curve in a single number between 0 and 1. Adding up the rectangles under the steps of the curve above gives 0.2 × 0.4 + 0.2 × 0.8 + 0.2 × 0.8 + 0.2 × 0.8 + 0.2 × 1.0 = 0.76.

AUC has a direct probabilistic meaning. It is the chance that a randomly chosen positive case scores higher than a randomly chosen negative case. Here there are 5 × 5 = 25 positive-negative pairs, and the positive outscores the negative in 19 of them, giving 19 ÷ 25 = 0.76, the same value. This is the same counting logic behind the Mann-Whitney statistic in non-parametric tests. An AUC of 0.5 means the score carries no information, and 1.0 means every positive outranks every negative.

Choosing a Threshold

AUC judges the score overall, but real use requires one threshold. A common starting point is the point that maximizes sensitivity + specificity − 1 (Youden's index). In the example, the threshold 0.55 gives 0.8 − 0.2 = 0.6, the best of the ten. Yet the right choice depends on costs. A screening test for a serious but treatable disease usually favors high sensitivity, accepting false alarms so that few cases are missed; a test that triggers an invasive or costly follow-up may favor high specificity instead.

Connection to Hypothesis Testing

These ideas mirror those in hypothesis testing. A false positive plays the role of a Type I error and a false negative that of a Type II error, so sensitivity is the same quantity as statistical power, and 1 − specificity corresponds to the significance level.

Common Mistakes

Quoting accuracy alone. When the condition is rare, a test that labels everyone negative is highly accurate and useless. At 1% prevalence it would score 99% accuracy while missing every case.

Confusing sensitivity with PPV. Sensitivity conditions on truly having the condition; PPV conditions on testing positive. Swapping them is the base rate error in another form.

Applying predictive values from one population to another. PPV and NPV change with prevalence, so figures from a specialist clinic do not carry over to general screening.

Judging by AUC when costs are lopsided. A high AUC says the score ranks cases well, not that any particular threshold makes good decisions for the problem at hand.

What is a good AUC?

There is no universal cutoff, as it depends on the field and the stakes. As rough guidance, 0.5 is no better than chance, values around 0.7 to 0.8 are often considered acceptable, and values above 0.9 are considered excellent, though very high AUCs on training data can signal overfitting.

Are sensitivity and recall the same thing?

Yes. Machine learning calls the true positive rate recall, and calls PPV precision. The formulas are identical to the medical versions.

Why can't a test have both perfect sensitivity and perfect specificity?

It can, if the scores of the two groups do not overlap at all, which produces an AUC of 1.0. In practice the groups overlap, so moving the threshold to gain one quality costs the other.

Does the ROC curve depend on prevalence?

No. Because sensitivity and specificity are each measured within a single group, the ROC curve and AUC are unaffected by how common the condition is. Predictive values, by contrast, shift with prevalence.