The Kolmogorov-Smirnov Test: Comparing a Distribution's Whole Shape

The Shapiro-Wilk test mentions the Kolmogorov-Smirnov test as an alternative way to check whether data follows a particular distribution, without explaining how it actually works. The Kolmogorov-Smirnov (K-S) test takes a genuinely different approach: rather than a specialized formula built around order statistics, it directly measures the largest gap between a sample's own cumulative distribution and a theoretical one — a simple, intuitive idea that turns out to generalize far beyond just testing for normality.

The Empirical CDF and the K-S Statistic

The empirical CDF is the step-function version of a CDF built directly from sample data: it jumps by 1/n at each sorted data point, reaching 1 once the largest value is passed. The Kolmogorov-Smirnov statistic, D, is the single largest vertical gap between this empirical CDF and the theoretical CDF the null hypothesis proposes:

A large D means the sample's actual distribution departs substantially from the proposed theoretical one somewhere along the range; a small D means the two track each other closely everywhere.

Worked Example

Problem: Using the same 6 exam scores from the Shapiro-Wilk test worked example — 62, 68, 71, 75, 78, 95, with sample mean 74.83 and standard deviation 11.34 — test whether the data is consistent with a Normal(74.83, 11.34) distribution using the K-S test.

For each sorted data point, compute the theoretical normal CDF, and compare it against the empirical CDF both just before and just after that point (since the empirical CDF jumps discontinuously):

The largest gap of all occurs at x=78: the theoretical CDF gives 0.610, while the empirical CDF has already jumped to 0.833 (5 of 6 points are at or below 78). D = |0.833 − 0.610| = 0.223. For n=6, this D statistic corresponds to a p-value of approximately 0.868 — well above 0.05, so the test fails to reject the hypothesis that the data comes from this normal distribution, the same conclusion the Shapiro-Wilk test reached on this identical dataset (p≈0.538 there).

The same conclusion is visible without computing a p-value at all: the exact critical value of D for n=6 at α=0.05 is 0.519 (see the Kolmogorov-Smirnov critical values table), and the observed D=0.223 falls well short of it.

Why the Two Tests Agree Here — and When They Might Not

Both tests reaching the same non-significant conclusion on this dataset isn't a coincidence specific to these numbers, but it's also not guaranteed in general — the two tests are built on genuinely different logic (Shapiro-Wilk on a weighted correlation between order statistics and their expected normal values; K-S on the single largest CDF gap), so they can occasionally disagree, especially for departures from normality concentrated in a way one test's specific method is more or less sensitive to. Shapiro-Wilk is generally regarded as having stronger power for detecting departures from normality specifically; the K-S test's real strength is its generality — the same test statistic and logic work for testing against any theoretical distribution, not just the normal distribution, without needing distribution-specific coefficient tables the way Shapiro-Wilk does. This generality is precisely why the K-S test remains popular well beyond normality checking, even in fields where a specialized, more powerful test for the specific distribution in question already exists.

The Two-Sample Version

Beyond comparing one sample against a theoretical distribution, the K-S test extends naturally to comparing two independent samples' empirical CDFs directly against each other, testing whether they could plausibly have come from the same underlying distribution — without needing to specify what that distribution actually is. This two-sample version is a genuinely useful, fully non-parametric alternative to comparing two groups, related in spirit to the rank-based comparison covered in non-parametric tests, but sensitive to any kind of distributional difference (not just a shift in typical value) between the two groups.

Real-World Uses

Model validation in simulation and finance: the K-S test is a standard check for whether simulated or modeled data (a random number generator's output, a fitted financial return model) genuinely matches its intended theoretical distribution, precisely because the test isn't limited to checking against the normal distribution specifically.

Comparing two datasets without assuming a distribution: the two-sample version is widely used whenever two groups need to be compared on their overall distribution shape — not just their average — such as comparing two sensors' full measurement distributions, or two time periods' full data distributions, when no particular theoretical shape can safely be assumed for either.

Goodness-of-fit checks in scientific research: across physics, biology, and other quantitative sciences, the K-S test is a standard tool for checking whether experimental data is consistent with a proposed theoretical model's predicted distribution.

Common Mistakes

Estimating the theoretical distribution's parameters from the same sample being tested, without adjustment. The classic K-S test's reference p-values assume the theoretical distribution's parameters are known in advance, not estimated from the same data — using sample-estimated parameters (as in the worked example above) technically requires an adjusted version (sometimes called the Lilliefors correction) for fully accurate p-values, though the basic K-S statistic itself is still commonly reported as an approximate guide. This is exactly the situation the worked example above runs into, since the mean and standard deviation used for the theoretical comparison were themselves estimated from the same 6 data points being tested.

Treating the K-S test as automatically more or less trustworthy than Shapiro-Wilk. Neither test is universally superior — Shapiro-Wilk tends to have better power specifically for normality testing, while the K-S test's flexibility to test against any distribution is its own distinct advantage, depending on what's actually being checked.