The Runs Test: Checking Whether a Sequence Is Actually Random

Most of the tests covered elsewhere on this site ask whether two groups differ, or whether a variable predicts another. The runs test (also called the Wald-Wolfowitz runs test) asks a different, more basic question: given a sequence of just two possible outcomes — pass/fail, heads/tails, above/below the median — does the order those outcomes appear in look random, or does it show a pattern a truly random process wouldn't produce?

What Counts as a "Run"

A run is a maximal streak of consecutive identical outcomes. The sequence P P P F F P P P P F F F P P F has 6 runs: PPP, FF, PPPP, FFF, PP, F. Notice that the runs test only looks at how the outcomes are ordered — it doesn't care that there are 9 P's and 6 F's overall, only whether those 9 P's and 6 F's are arranged the way a random shuffle would arrange them.

Why the Count of Successes Alone Isn't Enough

Two sequences can have the exact same number of each outcome and still look completely different. PPPPPPPP FFFF (8 P's then 4 F's, 2 runs) and PFPFPFPF PFPF (8 P's and 4 F's alternating, 12 runs) both have 8 P's and 4 F's — a simple count or proportion can't tell them apart. But neither looks like a random shuffle would: the first clumps everything together, and the second alternates far more often than chance would produce. The runs test is built specifically to catch exactly this kind of pattern.

The Exact Null Distribution

Given n₁ items of one kind and n₂ of the other, arranged in one of the C(n₁+n₂, n₁) equally likely orderings, the exact probability of observing exactly r runs is known in closed form (Wald & Wolfowitz, 1940):

Because this distribution is exact rather than approximated, exact lower and upper critical values exist for any n₁ and n₂ — see the runs test critical values table. Reject the hypothesis of randomness if the observed number of runs falls at or below the lower critical value (too few — clumping) or at or above the upper critical value (too many — alternating too often).

Worked Example: A Quality-Control Line

Problem: A quality inspector checks 12 consecutive items off a production line and records Pass (P) or Fail (F) for each: P P P P P P P P F F F F. That's 8 passes followed by 4 fails — 2 runs. Is this consistent with the failures occurring randomly throughout the shift, or does it look like something changed partway through?

Here n₁=8 (passes), n₂=4 (fails), and the observed number of runs is R=2. From the runs table, at α=0.05 with n₁=8 and n₂=4, the critical values are lower=3 and upper=10. Since the observed R=2 is at or below the lower critical value of 3, the result is statistically significant: this pattern is unlikely to have arisen from failures being randomly scattered through the 12 items. The clump of 4 fails at the very end is exactly the kind of shift-partway-through-the-shift pattern the runs test is designed to catch — worth investigating (a tool wearing down, a raw material batch change) rather than dismissing as ordinary variation.

Real-World Uses

Quality control: checking whether defects are scattered randomly through a production run or clustered in time, which would point to an assignable cause worth investigating rather than routine random variation — the same underlying idea behind the non-random-pattern rules used in control charts.

Randomness testing: checking whether a sequence generated by a random number generator, a coin-flip simulation, or a shuffling algorithm actually behaves like a genuinely random sequence, rather than showing hidden streakiness or an unnatural tendency to alternate.

Regression diagnostics: applied to the signs of a regression's residuals in time order, the runs test can catch a kind of unmodeled pattern (autocorrelation) that a simple residual plot might not make obvious, complementing tools like a Q-Q plot that check a different assumption (normality).

Common Mistakes

Confusing "too few runs" with "too many runs." Both are departures from randomness, but they mean opposite things: too few runs means clumping (a trend, a shift, a burst), while too many runs means suspiciously frequent alternation (overcorrection, an oscillating process). Always check which side of the critical range the observed count falls on before interpreting the result.

Applying it to more than two categories. The classic runs test is built for exactly two outcome types. Data with three or more categories needs to be reduced to two first — for example, testing values against their median and recording only above/below — or a different, dedicated method entirely.

Treating "not significant" as proof of randomness. As with any hypothesis test, failing to reject the null hypothesis of randomness means the data isn't strong enough evidence against it — it doesn't positively prove the sequence is random, especially with a small sample size where the test has limited power to detect real patterns.