What makes an estimator good

The sample mean turned out to be centred on the population mean with a spread of $σ / \sqrt{n}$ , which is reassuring but does not say it was the right choice. The median would also have been available, and so would the midpoint of the range, and for parameters with no obvious sample analogue there is no default at all. This lesson sets up the criteria that decide between candidates.

An estimator is a rule

Keep two things apart. An estimator is a rule: a function that turns any sample into a number, written $\hat{θ} = T (X_{1}, dots, X_{n})$ with capital letters because its inputs are random. An estimate is what that rule produces from the particular sample in hand, a single number with nothing random about it.

Every property discussed below belongs to the rule, not to the estimate. Saying that $\overline{X}$ is unbiased is a statement about what happens across all possible samples; it says nothing about whether $\overline{x} = 5.448$ is close to the truth in this instance, and it cannot, because the truth is unknown. This is the same shift the first lesson described, from claims about one conclusion to claims about a procedure, and it is what makes the subject tractable.

There are usually many candidate rules. For a symmetric population the mean, the median and the midrange all estimate the same centre. For the maximum of a uniform distribution on $[0, θ]$ , the sample maximum and twice the sample mean both estimate $θ$ . Choosing between them requires a definition of good.

Bias

The bias of an estimator is how far its sampling distribution is centred from the truth:

Bias (\hat{θ}) = E [\hat{θ}] - θ

An estimator with zero bias for every value of $θ$ is unbiased. The previous lesson showed $E [\overline{X}] = μ$ exactly, so the sample mean is unbiased for the population mean, at every sample size and for every population with a finite mean.

Bias is not the same as error. An unbiased estimator can be badly wrong on any given sample, and it is unbiased only in the sense that the errors cancel across repetitions. Conversely a biased estimator can be reliably close. The word carries a moral weight in ordinary English that it does not have here, and treating unbiasedness as a requirement rather than a property is the mistake this lesson is arranged to prevent.

Unbiasedness also fails to survive transformation. If $\hat{θ}$ is unbiased for $θ$ , then $g (\hat{θ})$ is generally biased for $g (θ)$ , because expectation does not pass through a nonlinear function. The sample variance $s^{2}$ is unbiased for $σ^{2}$ , and yet $s$ is biased low for $σ$ : for a normal population with $n = 10$ , $E [s] = 0.9727 σ$ , so the usual standard deviation is systematically 2.7 per cent too small. At $n = 25$ the factor is 0.9896 and at $n = 100$ it is 0.9975, so the problem fades, but it never vanishes and almost nobody corrects for it.

Where the $n - 1$ comes from

This is the derivation everyone is promised and rarely shown. Consider the sum of squared deviations from the sample mean, $SS = \sum_{i = 1}^{n} (X_{i} - \overline{X})^{2}$ , and rewrite each deviation by inserting the true mean:

X_{i} - \overline{X} = (X_{i} - μ) - (\overline{X} - μ)

Square and sum over $i$ . The cross term is $- 2 (\overline{X} - μ) \sum_{i} (X_{i} - μ)$ , and $\sum_{i} (X_{i} - μ) = n (\overline{X} - μ)$ , so the cross term is $- 2 n (\overline{X} - μ)^{2}$ , while the last term sums to $n (\overline{X} - μ)^{2}$ . Two of them cancel one, leaving the identity

\sum_{i = 1}^{n} (X_{i} - \overline{X})^{2} = \sum_{i = 1}^{n} (X_{i} - μ)^{2} - n (\overline{X} - μ)^{2}

Now take expectations. Each $E [(X_{i} - μ)^{2}]$ is $σ^{2}$ by definition, giving $n σ^{2}$ for the first sum. And $E [(\overline{X} - μ)^{2}]$ is the variance of the sample mean, which the previous lesson found to be $σ^{2} / n$ , so the second term has expectation $n (σ^{2} / n) = σ^{2}$ . Therefore

E [SS] = n σ^{2} - σ^{2} = (n - 1) σ^{2}

Dividing $SS$ by $n$ gives an expectation of $\frac{n - 1}{n} σ^{2}$ , which is too small; dividing by $n - 1$ gives exactly $σ^{2}$ . That is the whole justification, and the identity above shows precisely where the loss comes from: deviations are measured from $\overline{X}$ , which sits closer to the data than $μ$ does by exactly $(\overline{X} - μ)^{2}$ per observation on average.

The usual verbal gloss, that one degree of freedom is used up estimating the mean, is a summary of that algebra rather than an argument. It does generalise: when $k$ parameters are estimated before computing residuals, the divisor becomes $n - k$ , which is why the regression lesson divides by $n - 2$ .

Example. A sample of 5 values from a population with $σ^{2} = 100$ is taken, and the analyst divides the sum of squared deviations by 5 instead of 4. What does that estimator average out to?

By the result above, $E [SS] = 4 \times 100 = 400$ , so dividing by 5 gives an expectation of 80. The estimator is biased low by 20 per cent of the true variance, and no amount of repeating the experiment removes it, since 20 per cent is the average behaviour.

Now you. For $n = 20$ , what fraction of $σ^{2}$ does the divide-by- $n$ estimator average out to?

Answer

$(n - 1) / n = 19 / 20 = 0.95$ , so it is 5 per cent too low. The bias is $- σ^{2} / n$ , which shrinks as $n$ grows and is exactly why the two divisors are indistinguishable in large samples.

Variance and mean squared error

Centring is only half of the question. An estimator that is unbiased but wildly variable is worse than a slightly off-centre one that barely moves. The measure combining both is the mean squared error,

MSE (\hat{θ}) = E [(\hat{θ} - θ)^{2}] = Var (\hat{θ}) + Bias (\hat{θ})^{2}

and the decomposition is worth deriving, since it is three lines. Write $\hat{θ} - θ = (\hat{θ} - E [\hat{θ}]) + (E [\hat{θ}] - θ)$ . Squaring gives three terms; the first has expectation $Var (\hat{θ})$ , the third is a constant equal to the squared bias, and the cross term has expectation zero because $E [\hat{θ} - E [\hat{θ}]] = 0$ . So MSE splits cleanly into a spread part and an off-centre part, and minimising it means trading one against the other.

That trade is real, not hypothetical. For a normal sample, consider estimating $σ^{2}$ by $c \cdot SS$ for a constant $c$ . Choosing $c = 1 / (n - 1)$ gives the unbiased $s^{2}$ , with $MSE = 2 σ^{4} / (n - 1)$ . Choosing $c = 1 / (n + 1)$ introduces a small downward bias and minimises the MSE. At $n = 10$ the unbiased choice has MSE $0.2222 σ^{4}$ and the biased one has $0.1818 σ^{4}$ , eighteen per cent lower. The biased estimator is closer to the truth on average, by the squared-error standard, and the unbiased one is still the one everybody uses, largely because unbiasedness composes nicely across later calculations.

Example. Draws come from a uniform distribution on $[0, θ]$ . Two estimators of $θ$ are $2 \overline{X}$ and $\frac{n + 1}{n} M$ , where $M$ is the sample maximum. Both are unbiased. Compare their variances at $n = 10$ .

The uniform has mean $θ / 2$ and variance $θ^{2} / 12$ , so $Var (2 \overline{X}) = 4 (θ^{2} / 12) / n = θ^{2} / (3 n)$ , which is $θ^{2} / 30 = 0.0333 θ^{2}$ . The maximum has $E [M] = n θ / (n + 1)$ and the rescaled version has variance $θ^{2} / (n (n + 2))$ , which is $θ^{2} / 120 = 0.00833 θ^{2}$ . The maximum-based estimator has a quarter of the variance, because the largest observation carries nearly all the information about an upper limit while the other nine contribute almost nothing.

Now you. How does the advantage change with $n$ ? Compare the two variances as formulas.

Answer

The ratio is $\frac{θ^{2} / (3 n)}{θ^{2} / (n (n + 2))} = \frac{n + 2}{3}$ , so the advantage grows without limit. The mean-based estimator improves as $1 / n$ while the maximum-based one improves as $1 / n^{2}$ , which is a different and much faster rate.

Consistency

A minimum requirement is that more data should help. An estimator is consistent if $\hat{θ}$ converges in probability to $θ$ as $n$ grows: for any tolerance $ε$ , the probability of missing by more than $ε$ goes to zero.

The sample mean is consistent by the weak law of large numbers, proved in the probability course, and the argument is available directly from Chebyshev's inequality: $P (| \overline{X} - μ | \geq ε) \leq σ^{2} / (n ε^{2})$ , which goes to zero. More generally, any estimator whose bias and variance both tend to zero is consistent, since its MSE tends to zero.

Consistency is weak, and it is worth knowing how weak. It says nothing about any finite sample size, so an estimator can be consistent and useless at $n = 500$ . It is also easy to satisfy: adding $10^{6} / n$ to a consistent estimator leaves it consistent while making it absurd for realistic samples. Its real value is as a filter: an inconsistent estimator is disqualified, because it does not converge on the answer even with unlimited data. The first lesson's biased thermometer is exactly that, converging with beautiful precision on the wrong number.

Efficiency, and the reason the mean usually wins

Among unbiased estimators, the one with the smallest variance is called efficient, and comparing two is done by the ratio of their variances. For a normal population the sample mean has variance $σ^{2} / n$ , while the sample median has variance approaching $π σ^{2} / (2 n) = 1.571 σ^{2} / n$ for large $n$ . The relative efficiency of the median is therefore $2 / π = 0.637$ : a median computed from 1000 observations carries the information of a mean computed from 637.

There is a floor on how well any unbiased estimator can do, given by the Cramér-Rao bound, which says that the variance of an unbiased estimator is at least the reciprocal of the Fisher information in the sample. The next lesson introduces that quantity properly. What matters here is that the bound exists, that the sample mean attains it for normal data, and that "the mean is best" is therefore a theorem under stated assumptions rather than a habit.

Change the assumptions and the answer changes. For a Laplace population, whose density is proportional to $e^{- | x - μ |}$ and which has heavier tails than the normal, the median is the efficient estimator and the mean is the wasteful one. For a population with occasional gross errors, the mean's variance can be arbitrarily bad while the median's is not. The second lesson's advice, mean for clean symmetric data and median for contaminated data, is the practical statement of exactly this.

Example. An analyst has 1000 observations from a normal population and reports the median rather than the mean, to be safe against outliers. What has that caution cost, expressed in sample size?

The median's variance is $π / 2$ times the mean's, so the analysis has the precision of $1000 \times 2 / π = 637$ observations. The 363 discarded observations are the insurance premium, and it is worth paying only if outliers are a real prospect.

Now you. A study reports means and wants the same precision the median would give from 2000 observations. How many observations does it need?

Answer

The mean needs $2000 \times 2 / π = 1273$ observations to match a median computed from 2000. The comparison runs both ways: the efficient estimator needs fewer observations for the same precision.

What is still missing

There are now criteria: bias, variance, mean squared error, consistency and efficiency, and enough machinery to compare any two candidate rules. What there is not is a way to produce candidates. Every estimator used so far was guessed by analogy, replacing a population mean by a sample mean, a population maximum by a sample maximum, and that analogy runs out quickly. Nothing in it says how to estimate the rate of a Poisson process from interval counts, or the two parameters of a distribution at once, or a parameter that appears only inside an exponential.

What is needed is a general recipe that takes a probability model and returns an estimator, ideally one that is automatically consistent and efficient in large samples. That recipe exists and is the subject of the next lesson.

An estimator is a rule

Bias

Where the n-1 comes from

Variance and mean squared error

Consistency

Efficiency, and the reason the mean usually wins

What is still missing

Where the $n - 1$ comes from