The Kruskal-Wallis Test: ANOVA Without the Normality Assumption

ANOVA compares three or more group means, but it assumes the data in each group is roughly normally distributed with similar spread. When that assumption is shaky — small samples, skewed data, or a handful of extreme values — the Kruskal-Wallis test asks the same basic question (do these groups differ?) without needing normality at all, by working with ranks instead of raw values, exactly the way the Spearman correlation extends Pearson's r to ranks.

How It Works

Every observation from every group is pooled together and ranked from smallest to largest, ignoring which group it came from. If the groups truly come from the same distribution, their average ranks should end up similar just by chance. The test statistic H measures how far the groups' actual rank sums are from that chance expectation:

where N is the total number of observations across all groups, nᵢ is the size of group i, and Rᵢ is the sum of the ranks in group i. A larger H means the groups' rank sums are more spread apart than chance alone would produce.

Worked Example: Three Training Methods

Problem: A small pilot study compares three training methods on 9 employees (3 per method), measuring a skill score after training. Method A: 85, 88, 82. Method B: 91, 95, 89. Method C: 78, 75, 80. Do the methods differ?

Pooling and ranking all 9 scores (1 = lowest): 75→1, 78→2, 80→3, 82→4, 85→5, 88→6, 89→7, 91→8, 95→9. Group rank sums: R_A = 5+6+4 = 15, R_B = 8+9+7 = 24, R_C = 2+1+3 = 6.

H = [12/(9×10)] × (15²/3 + 24²/3 + 6²/3) − 3(10) = (12/90) × (75+192+12) − 30 = 0.1333 × 279 − 30 = 37.2 − 30 = 7.2.

With only 3 observations per group, the large-sample chi-square approximation isn't reliable — this calls for the exact Kruskal-Wallis critical values table. For n=(3,3,3), the exact critical value at α=0.01 is exactly 7.2 — meaning the observed H clears even the strictest of the three listed significance levels, with an exact p-value of 6/1680 ≈ 0.0036. The methods differ significantly; Method B's scores clearly rank higher than Method C's.

Exact vs. Approximate Critical Values

Under the null hypothesis of no group differences, H is only approximately chi-square distributed with k−1 degrees of freedom (k = number of groups) — this approximation gets better as sample sizes grow, but is unreliable for the small samples where a non-parametric test is often chosen in the first place. For k=3 with each group between 2 and 5 observations, the exact null distribution of H can be found directly, by literally listing every equally likely way the ranks could have landed in the groups, exactly the same brute-force logic behind the exact Spearman table and the exact runs test table elsewhere on this site. Beyond that range, the chi-square approximation with k−1 degrees of freedom is the standard, and accurate, fallback.

Real-World Uses

Small pilot studies: comparing a handful of treatment groups with only a few observations each, where normality can't reasonably be checked or assumed, and where the exact test avoids relying on an approximation that hasn't kicked in yet.

Skewed measurements: reaction times, income, wait times, and similar right-skewed data across several groups, where ANOVA's normality assumption is routinely violated but a rank-based comparison remains valid.

Ordinal outcomes: comparing three or more groups on a rating scale (satisfaction from 1 to 5, a Likert-type item) where the values are ordered but not meaningfully averaged, so ranking is a more natural comparison than a mean.

Common Mistakes

Interpreting a significant result as "the groups have different medians." Kruskal-Wallis is a test of whether the distributions differ, driven mainly by rank location — with very differently shaped distributions (not just shifted, but different spread or skew) a significant H doesn't guarantee the medians themselves differ.

Not following up with pairwise comparisons. Just like ANOVA, a significant Kruskal-Wallis result only says "the groups aren't all the same" — it doesn't identify which pairs differ. A follow-up procedure (such as pairwise Mann-Whitney tests with a correction for multiple comparisons, echoing the multiple comparisons problem) is needed to pin that down.

Using the chi-square approximation with tiny samples. With only 2-5 observations per group, the exact table gives a real answer where the chi-square approximation can be noticeably off — always prefer the exact critical value when it's available for your sample sizes.