McNemar's Test: Comparing Paired Yes/No Outcomes

Some yes/no comparisons involve the same people twice: the same customers asked before and after an advertising campaign, the same patients tested by two different diagnostic methods, the same students marked pass or fail on two exams. Because each person appears in both measurements, the two sets of answers are not independent, and an ordinary comparison of two proportions would ignore that. McNemar's test is the standard tool for paired binary data. It is the yes/no counterpart of the paired t-test, and its central idea is that only the people whose answer changed carry any information about whether things shifted.

Setting Up the Table

Each subject gives a pair of answers, so each falls into one of four cells. The table crosses the first answer with the second:

	After: Yes	After: No
Before: Yes	a (stayed Yes)	b (Yes to No)
Before: No	c (No to Yes)	d (stayed No)

The cells a and d are concordant pairs: the person gave the same answer both times, so they say nothing about a change. The cells b and c are discordant pairs, where the answer switched, and they are the only cells that matter. Under the null hypothesis that the campaign had no net effect, a switch from Yes to No is exactly as likely as a switch from No to Yes, so among the switchers, b and c should be roughly equal.

Worked Example

Problem: 100 shoppers are asked "Would you buy this product?" before and after seeing an advertising campaign. Is there evidence that the campaign changed buying intent?

	After: Yes	After: No	Total
Before: Yes	40	10	50
Before: No	25	25	50
Total	65	35	100

Before the campaign, 50 shoppers said Yes; afterward, 65 did, an increase of 15 percentage points. Of the 100 shoppers, 65 gave the same answer twice, so the story lives in the 35 who switched: b = 10 went from Yes to No, and c = 25 went from No to Yes.

Step 1 — Hypotheses. H₀: switches in each direction are equally likely (the two marginal proportions are equal). H₁: they are not (a two-sided test, as discussed in one-tailed vs. two-tailed tests).

Step 2 — The test statistic. McNemar's chi-square statistic uses only b and c:

The "−1" is a continuity correction, which adjusts for using a smooth chi-square curve to approximate counts. The statistic is compared with a chi-square distribution with 1 degree of freedom, as in the chi-square test. The p-value is about 0.018. Without the correction, χ² = 15² ÷ 35 ≈ 6.43 and p ≈ 0.011.

Step 3 — Conclusion. At the 5% level, reject H₀. Among shoppers who changed their answer, significantly more moved toward buying than away from it.

The Exact Version

When b + c is small (a common rule of thumb is fewer than about 25), the chi-square approximation is unreliable and the exact test is preferred. It treats the b + c discordant pairs as trials of a fair coin under H₀: each switch is equally likely to go either way, so c follows a binomial distribution with n = 35 and p = 0.5. The chance of seeing 25 or more switches in the "No to Yes" direction is about 0.0083, and doubling for a two-sided test gives an exact p-value of 0.0167. That is very close to the corrected chi-square result of 0.018, confirming the approximation works well here. This exact form is the sign test applied to the discordant pairs, a member of the family in non-parametric tests.

Estimating the Size of the Change

A p-value says a change is unlikely to be chance; it does not say how large it is. The difference between the two marginal proportions, (c − b) ÷ n = (25 − 10) ÷ 100 = 0.15, estimates the change directly. Its standard error for paired data is √(b + c − (c − b)² ÷ n) ÷ n = √(35 − 2.25) ÷ 100 ≈ 0.057, giving an approximate 95% confidence interval of 0.15 ± 1.96 × 0.057, or about 0.04 to 0.26. The interval excludes zero, consistent with the test, but it is wide, so the true shift could be modest or large. Another common summary is the matched-pairs odds ratio, c ÷ b = 25 ÷ 10 = 2.5: among switchers, a move toward Yes was two and a half times as common as a move toward No. Reporting an interval like this alongside the p-value follows the advice in confidence intervals.

Why the Pairing Matters

Suppose the two answer sets were wrongly treated as independent samples, comparing 50 out of 100 with 65 out of 100 using a two-proportion z-test. The pooled proportion is 0.575, the standard error is √(0.575 × 0.425 × 2 ÷ 100) ≈ 0.070, and z ≈ 2.15, giving p ≈ 0.032. That still passes at 5%, but it is nearly twice the McNemar p-value of 0.018. The independent-samples test throws away the fact that each shopper serves as their own control, so it counts the stable pairs (the 65 who did not change) as noise. McNemar's test removes that noise by looking only at the switchers. The stronger the pairing, the bigger the gain, and in other datasets the wrong test can miss a real effect entirely.

Real-World Uses

Comparing diagnostic tests: when two tests are run on the same patients, McNemar's test checks whether one detects the condition more often than the other, using only the patients on whom the tests disagree.

Before-and-after surveys: opinion, awareness, or intention measured on the same respondents before and after an event or campaign.

Comparing two classifiers: in machine learning, McNemar's test can compare two models evaluated on the same test set by counting the cases one gets right and the other gets wrong, as an alternative to relying on accuracy differences alone (see k-fold cross-validation for another way to evaluate models).

Common Mistakes

Using a chi-square test of independence. The standard test on a 2 × 2 table assumes the two groups are separate people. On paired data it answers the wrong question and can badly misjudge the evidence.

Ignoring the concordant pairs entirely when reporting. The test itself does not use them, but a report should still show the full table and the marginal percentages, so readers see the practical size of the change.

Using the chi-square approximation with too few discordant pairs. With b + c below about 25, use the exact version instead.

Applying it to more than two measurements. With three or more paired yes/no measurements on the same subjects, McNemar's test no longer applies, and the related Cochran's Q test is used instead.

When should I use McNemar's test instead of a paired t-test?

Use McNemar's test when each measurement is binary, a yes/no or pass/fail outcome. The paired t-test is for numeric measurements such as scores or weights taken twice on the same subjects. The choosing a statistical test guide covers the full decision.

Why does the test ignore the pairs that did not change?

Because they provide no evidence about direction. Someone who said Yes both times gives no information on whether Yes became more or less common. Only the people who switched can tell the two directions apart.

What does the null hypothesis actually say?

It says the two marginal proportions are equal, which is equivalent to saying switches in each direction are equally likely, so that b and c are expected to be equal apart from chance.

Is the continuity correction always needed?

It is standard in many textbooks because it makes the approximation closer to the exact result for modest counts, but some software offers both versions. When b + c is small, the exact test avoids the question entirely.