Bootstrap Sampling: Confidence Intervals Without a Formula

Most confidence intervals rely on a formula that assumes a specific distribution shape — normal, t, or otherwise. But what if you want a confidence interval for something with no clean formula, like a median, or a more complicated statistic entirely? Bootstrap sampling sidesteps the need for a formula altogether, using the computer's raw power to estimate uncertainty directly from the data itself.

The Core Idea

Bootstrapping extends the same underlying goal as confidence intervals — quantifying how much a sample statistic might vary — to situations where the traditional formula-based approach either doesn't exist or relies on assumptions the data doesn't meet.

Given an original sample of size n, a bootstrap sample is created by drawing n observations FROM THAT SAME SAMPLE, with replacement — meaning some original values may appear multiple times in a given bootstrap sample, and others may not appear at all. Calculate your statistic of interest (mean, median, correlation, or anything else) on this bootstrap sample. Repeat this process thousands of times, each time drawing a fresh bootstrap sample and recalculating the statistic. The resulting collection of thousands of statistic values — the "bootstrap distribution" — approximates the sampling distribution that statistic would have if you could actually draw many real samples from the true population, which in practice you almost never can.

Building a Bootstrap Confidence Interval

Once you have, say, 10,000 bootstrap estimates of a statistic, the simplest bootstrap confidence interval (the "percentile method") just takes the 2.5th and 97.5th percentiles of that bootstrap distribution directly as the bounds of a 95% confidence interval — no formula, no distributional assumption required, just direct percentiles of the simulated distribution.

Worked Example (Conceptual)

Problem: A sample of 8 patients' recovery times (days): 5, 7, 6, 9, 20, 6, 8, 7. The median is 7 days, but with an outlier (20) present and only 8 observations, no simple formula exists for a reliable median confidence interval. How does bootstrapping solve this?

Draw a bootstrap sample of 8 values, WITH replacement, from these same 8 numbers — for instance, one such resample might be: 7, 6, 6, 9, 20, 7, 5, 6 (note 6 appears three times, 8 doesn't appear at all — both allowed under sampling with replacement). Calculate its median. Repeat this 10,000 times, recording each resample's median. The spread of these 10,000 median values — taking the 2.5th and 97.5th percentiles — directly gives a 95% confidence interval for the true median, without ever needing a formula for "the standard error of a median," which doesn't have a simple closed form the way the mean's does.

When Bootstrapping Is Especially Useful

Bootstrapping earns its keep specifically when a statistic has no simple, well-established formula for its standard error or confidence interval — medians, correlation coefficients in some contexts, ratios of two statistics, and many machine learning model performance metrics all fall into this category. It's also useful when a formula technically exists but relies on distributional assumptions (like normality) that the actual data clearly violates, since the bootstrap approach makes no such assumption about the underlying population's shape.

What Bootstrapping Doesn't Fix

Bootstrapping resamples from your ORIGINAL sample — it cannot invent information that wasn't in that sample to begin with. If the original sample is small, biased, or unrepresentative of the true population, every bootstrap resample inherits that same limitation; bootstrapping estimates uncertainty around what your actual sample shows, it doesn't correct a fundamentally flawed or non-random original sample. This is a genuinely important limit: bootstrapping is not a substitute for collecting a larger or better-designed original sample, and treating it as one is a real misunderstanding of what the method actually does.

Real-World Uses

Machine learning: bootstrap resampling underlies "bagging" (bootstrap aggregating), the technique behind random forest models — each individual tree in the forest is trained on a separate bootstrap sample of the original training data, and their predictions are then combined.

Estimating uncertainty for complex statistics: whenever a model's performance metric or a complicated derived statistic has no clean analytical formula for its confidence interval, bootstrapping provides a general-purpose, assumption-light alternative.

Medical and biological research: bootstrapping is commonly used to build confidence intervals for statistics from small clinical samples, where normality assumptions underlying traditional formulas are often questionable.

Common Mistakes

Treating bootstrapping as a fix for a small or biased sample. As covered above, this is the single most important limitation to understand — bootstrapping quantifies uncertainty within what your sample already shows; it cannot compensate for a sample that wasn't randomly or representatively collected in the first place.

Using too few bootstrap resamples. A bootstrap distribution built from only 50 or 100 resamples can be noticeably unstable — the same original data set can produce a visibly different confidence interval on separate runs with too few resamples, undermining the reproducibility of the result.

Applying the simple percentile method when a more refined bootstrap method is warranted. The basic percentile method shown above is the simplest bootstrap confidence interval, but more sophisticated variants (like the bias-corrected and accelerated, or "BCa," method) correct for known biases the simple percentile approach can have, particularly for skewed statistics — worth knowing this more refined option exists even though the basic version is a reasonable starting point for most everyday uses.

Bootstrapping vs. the Traditional Formula-Based Approach

For a statistic like the mean, where a well-established formula-based confidence interval already exists (covered in confidence intervals), bootstrapping and the traditional formula typically produce very similar results when the underlying assumptions of the formula-based approach genuinely hold. The real value of bootstrapping shows up precisely where those assumptions break down, or where no formula exists at all — it's not generally meant to replace formula-based methods across the board, but to extend confidence-interval construction to situations the traditional formulas can't easily handle.

A Second Worked Example: Bootstrapping a Correlation

Problem: A small sample of 10 paired observations shows a Pearson correlation of r=0.65. How can bootstrapping estimate a confidence interval for this correlation?

Draw a bootstrap sample of 10 PAIRS (keeping each x-y pair together, since correlation depends on the relationship between paired values, not either variable alone) with replacement from the original 10 pairs. Calculate the correlation of this resampled pair set. Repeat 10,000 times. The 2.5th and 97.5th percentiles of these 10,000 bootstrap correlation values give a 95% confidence interval for the true correlation — a calculation that has no simple, universally-agreed closed-form formula the way a mean's confidence interval does, making bootstrapping a genuinely useful alternative here rather than just a convenient one.