Unbiased Estimators and Mean Squared Error: What Makes an Estimate Good?

Every statistic calculated from a sample is really an estimator: a rule for guessing an unknown population quantity from limited data. The sample mean estimates the population mean, the sample variance estimates the population variance, and a sample proportion estimates a population proportion. Because different samples give different answers, an estimator is itself a random quantity with its own distribution. Judging whether an estimator is good means asking two questions about that distribution: is it centred on the right target, and how widely does it scatter?

Bias, Variance, and MSE

The bias of an estimator is the gap between its average value across all possible samples and the true parameter it targets. An estimator whose average equals the truth is unbiased. Its variance measures how much individual estimates scatter around their own average. The single number that combines both is the mean squared error, the average squared distance between the estimate and the truth:

A dartboard picture helps. Bias is how far the centre of your cluster of darts sits from the bullseye; variance is how spread out the cluster is. A tight cluster in the wrong place and a wide cluster centred correctly can both score poorly, and MSE penalizes each kind of failure.

Worked Example: An Exact Comparison

Problem: A tiny population contains just three values: 0, 3, and 6. Its mean is 3 and its population variance (dividing by the population size 3) is (9 + 0 + 9) ÷ 3 = 6. Samples of size n = 2 are drawn with replacement, so there are 3 × 3 = 9 equally likely samples. Compare two estimators of the variance: s² dividing by n − 1 = 1, and a version dividing by n = 2.

Sample	Sample mean	s² (divide by n − 1)	Divide by n
0, 0	0	0	0
0, 3	1.5	4.5	2.25
0, 6	3	18	9
3, 0	1.5	4.5	2.25
3, 3	3	0	0
3, 6	4.5	4.5	2.25
6, 0	3	18	9
6, 3	4.5	4.5	2.25
6, 6	6	0	0
Average	3	6	3

Averaging each column across all nine samples tells the story:

• The sample mean averages exactly 3, the true mean, so it is unbiased. Its variance across samples is 3, which equals σ² ÷ n = 6 ÷ 2, matching the standard error formula.
• The divide-by-(n − 1) variance averages exactly 6, the true variance, so it is unbiased.
• The divide-by-n variance averages only 3, half the truth. Its bias is 3 − 6 = −3. In general, its average is σ² × (n − 1) ÷ n, which for n = 2 is exactly half.

That last line is the entire reason for Bessel's correction: dividing squared deviations by n systematically underestimates the variance, because deviations are measured from the sample's own mean, which sits closer to the sample values than the true mean does. Dividing by n − 1 exactly cancels the shortfall. The full working of the calculator version is on the standard deviation calculator page.

Unbiased Is Not the Same as Most Accurate

Now compare accuracy using MSE. For the divide-by-(n − 1) estimator, the errors relative to the true 6 are −6 (in 3 samples), −1.5 (in 4 samples), and +12 (in 2 samples). Its MSE is (3 × 36 + 4 × 2.25 + 2 × 144) ÷ 9 = 405 ÷ 9 = 45, and since it is unbiased, that is all variance.

For the divide-by-n estimator, the errors are −6 (3 samples), −3.75 (4 samples), and +3 (2 samples). Its MSE is (3 × 36 + 4 × 14.0625 + 2 × 9) ÷ 9 = 182.25 ÷ 9 = 20.25. It has a bias of −3 (contributing 9) and a variance of 11.25, and 9 + 11.25 = 20.25.

The biased estimator has less than half the MSE of the unbiased one. Dividing by a larger number shrinks all the estimates, which reduces their scatter by more than the bias costs. This is a small, extreme example, but the effect is real in general. For normally distributed data, the divide-by-(n − 1) estimator has MSE 2σ⁴ ÷ (n − 1), while the divide-by-n version has MSE (2n − 1)σ⁴ ÷ n². At n = 10, that is 0.222σ⁴ against 0.190σ⁴, so the biased version is more accurate on average, and dividing by n + 1 would do better still.

So Why Divide by n − 1?

Because unbiasedness has practical value even when it does not minimize MSE. Formulas for confidence intervals and t-tests are built on the unbiased version, which makes results consistent and comparable across studies. As n grows, the difference between dividing by n and n − 1 fades, and the MSE advantage of the biased version shrinks with it. Unbiasedness is a convenient, principled default, not a proof of optimality.

The same trade-off drives a good deal of modern statistics. Ridge and lasso regression deliberately accept a little bias to buy a large reduction in variance, and the bias-variance trade-off discussed in statistics in machine learning is this same MSE decomposition applied to prediction.

A Subtle Trap: Unbiased Variance, Biased Standard Deviation

Taking the square root of an unbiased variance estimate does not give an unbiased standard deviation. In the example, the nine values of s are 0 (three times), √4.5 ≈ 2.12 (four times), and √18 ≈ 4.24 (twice), averaging about 1.89, well below the true σ = √6 ≈ 2.45. Because the square root curves downward, s underestimates σ on average even though s² is exactly right. The gap shrinks with larger samples and is usually ignored in practice.

Consistency and Efficiency

Two more properties round out the checklist. An estimator is consistent if it converges to the true value as the sample size grows, which is the guarantee behind the law of large numbers. Both variance estimators above are consistent, even though one is biased in small samples. An estimator is efficient if it has the smallest variance among comparable estimators. For normally distributed data, the sample mean beats the sample median as an estimator of the centre: the median's variance is about π ÷ 2 ≈ 1.57 times larger in large samples, so it needs roughly 57% more data for the same precision.

Real-World Uses

Survey design: pollsters choose estimators, such as weighted proportions, that are unbiased or nearly so, then report the standard error to convey the remaining scatter.

Model fitting: maximum likelihood estimation produces estimators that are consistent and efficient in large samples but not always unbiased in small ones. The maximum likelihood estimate of a normal variance divides by n, exactly the biased version above.

Machine learning: choosing a model's complexity is largely a search for the point where bias and variance together give the lowest prediction error.

Common Mistakes

Equating unbiased with accurate. An unbiased estimator can be wildly scattered. Bias says nothing about variance, and a single estimate from a high-variance estimator can land far from the truth.

Assuming bias means the estimate is wrong. Bias is a property of the procedure averaged over many samples, not of any single estimate. A biased estimator can produce estimates closer to the truth, on average, than an unbiased one.

Forgetting that bias depends on sample size. Many biased estimators become negligibly biased as n grows, so the practical concern is largest in small samples.

Applying the n − 1 rule to a full population. If the data are the entire population rather than a sample, dividing by N is correct, because there is nothing to estimate.

Is the sample mean always an unbiased estimator?

For the population mean, yes, provided the sample is drawn without systematic selection problems. Bias from a flawed sampling process, such as those in types of bias in research, is a separate problem that no formula can repair.

Can bias be estimated from a single sample?

Sometimes. Resampling methods such as the bootstrap can estimate an estimator's bias and variance directly from the data, which is useful for statistics with no simple formula.

Which is better, low bias or low variance?

Neither wins in general. MSE combines both, so the best estimator is the one that minimizes total error for the problem at hand, which sometimes means accepting a little bias for a large drop in variance.

Why is the sample proportion unbiased?

A sample proportion is a mean of 0/1 outcomes, so its average over all possible samples equals the true proportion. Its variance is p(1 − p) ÷ n, which shrinks as the sample grows.