Confidence intervals

An estimate with a standard error beside it contains everything needed to say how precise it is, and almost nobody reads it that way. A confidence interval packages the same two numbers into a range, which is harder to misread and, as it turns out, still routinely misread. This lesson derives the construction, states exactly what it guarantees, and shows a case where the standard textbook interval does not deliver what it advertises.

From a point to a range

Suppose $\hat{θ}$ is approximately normal around $θ$ with standard error $σ_{\hat{θ}}$ , which the maximum likelihood theory of the previous lesson supplies for nearly any model. Then before the data is seen,

P (- 1.96 \leq \frac{\hat{θ} - θ}{σ_{\hat{θ}}} \leq 1.96) = 0.95

since the middle quantity is standard normal and $Φ (1.96) = 0.975$ . The inequalities can be rearranged so that $θ$ sits alone in the middle:

P (\hat{θ} - 1.96 σ_{\hat{θ}} \leq θ \leq \hat{θ} + 1.96 σ_{\hat{θ}}) = 0.95

The interval $\hat{θ} \pm 1.96 σ_{\hat{θ}}$ is the 95 per cent confidence interval. The quantity that made this work, $(\hat{θ} - θ) / σ_{\hat{θ}}$ , is called a pivot: a function of both the data and the parameter whose distribution does not depend on the parameter. Every interval in this course is built the same way, by finding a pivot, writing down a probability statement about it, and solving for the parameter.

The rearrangement is algebra, and the interpretation is where care is required, because the probability statement was made before the data arrived and the interval is written after.

What the 95 per cent is a statement about

The correct reading is about the procedure. If the study were repeated many times, with a fresh sample each time and the interval recomputed each time, 95 per cent of those intervals would contain the true $θ$ . That is the guarantee, and it is a guarantee about a rule, exactly as in the lesson on estimators.

The incorrect reading, and it is the one most people carry, is that there is a 95 per cent probability that $θ$ lies inside this particular interval. Once the data is in hand, the interval is a pair of fixed numbers and $θ$ is a fixed unknown constant, so $θ$ is either in it or not, and no probability between 0 and 1 applies. Saying "95 per cent confident" is a way of not saying "95 per cent probable", and the distinction is not pedantry: a reader who believes the second will happily interpret a 95 per cent interval that runs from 0.1 to 12 as though the middle of it were more probable than the ends, and the frequentist construction says nothing of the sort.

Two consequences follow directly. A 99 per cent interval is wider than a 95 per cent one, since a stricter coverage requirement needs more room: the multiplier moves from 1.96 to 2.576. And the confidence level is chosen by the analyst before looking, because choosing it afterwards to make an interval exclude some value destroys the property being claimed.

Gosset's t

The interval above used $σ_{\hat{θ}}$ , which for a mean is $σ / \sqrt{n}$ and contains the unknown population standard deviation. Substituting $s$ seems harmless and is not, for small $n$ : the quantity

T = \frac{\overline{X} - μ}{s / \sqrt{n}}

has a random denominator, so it varies more than a standard normal does, and using 1.96 gives intervals that are too narrow and cover less often than advertised.

William Gosset, a brewer at Guinness in Dublin working with samples of four or five barley plots, worked out the exact distribution of $T$ in 1908 and published it as "Student" because his employer treated statistical methods as a trade secret. For a normal population, $T$ follows the $t$ distribution with $n - 1$ degrees of freedom: symmetric, centred at zero, and with heavier tails than the normal, converging to the normal as the degrees of freedom grow.

The practical effect is entirely in the multiplier. For a 95 per cent interval it is 2.776 at 4 degrees of freedom, 2.262 at 9, 2.045 at 29, 1.979 at 129 and 1.960 in the limit. Beyond about 30 observations the correction is under 5 per cent and beyond 100 it is under 1 per cent, which is why large-sample work ignores it and small-sample work cannot. The interval is

\overline{x} \pm t_{0.975, n - 1} \frac{s}{\sqrt{n}}

The $t$ distribution assumes a normal population, and that assumption matters most exactly where the correction does, at small $n$ , since the central limit theorem has not had a chance to work. For $n$ under about 15 with visibly skewed data, neither the normal nor the $t$ interval is trustworthy.

A worked interval on real data

Body temperature is conventionally 37 °C, or 98.6 °F, a figure due to Carl Wunderlich's measurements in the 1860s. In 1992 Mackowiak and colleagues remeasured it, and the 130 readings tabulated from that work have $\overline{x} = 98.249$ °F and $s = 0.733$ °F.

The estimated standard error is $0.733 / \sqrt{130} = 0.0643$ °F. With 129 degrees of freedom the multiplier is 1.9785, so the margin is $1.9785 \times 0.0643 = 0.127$ °F and the interval is

98.249 \pm 0.127 = (98.122, 98.376) °F

which in Celsius is 36.734 to 36.876, centred on 36.805 °C. The conventional 98.6 °F lies well outside it: it is $(98.249 - 98.6) / 0.0643 = 5.46$ standard errors away, which is the kind of distance the normal curve makes essentially impossible by chance. The interval says the mean is near 36.8 °C and not 37 °C, and it says so with a precision of about a tenth of a degree Fahrenheit.

Example. A sample of 25 measurements has $\overline{x} = 12.4$ and $s = 3.0$ . Construct the 95 per cent confidence interval for $μ$ .

The estimated standard error is $3.0 / \sqrt{25} = 0.6$ . With 24 degrees of freedom the multiplier is 2.064, so the margin is 1.238 and the interval is $(11.16, 13.64)$ . Using 1.96 instead would give a margin of 1.176, about 5 per cent too narrow.

Now you. A sample of 10 has $\overline{x} = 50.0$ and $s = 4.0$ . Construct the 95 per cent interval.

Answer

The standard error is $4.0 / \sqrt{10} = 1.265$ , the multiplier at 9 degrees of freedom is 2.262, so the margin is 2.861 and the interval is $(47.14, 52.86)$ . At this sample size the $t$ multiplier is 15 per cent above 1.96, which widens the interval by the same 15 per cent.

Intervals for a proportion

For a proportion the same recipe gives the Wald interval,

\hat{p} \pm 1.96 \sqrt{\frac{\hat{p} (1 - \hat{p})}{n}}

using the standard error derived from the Fisher information in the previous lesson. With 240 supporters out of 400, $\hat{p} = 0.60$ , the standard error is 0.0245 and the interval is $(0.552, 0.648)$ .

This is the interval in every introductory course, and its actual coverage is not 95 per cent. Because the binomial is discrete and $\hat{p}$ appears in the standard error as well as at the centre, the true coverage can be computed exactly by summing binomial probabilities over the outcomes whose intervals contain $p$ . At $n = 20$ and $p = 0.3$ the coverage is 94.7 per cent, tolerable; at $n = 20$ and $p = 0.1$ it is 87.6 per cent, which means a nominal one-in-twenty failure rate is really one in eight. The failure is worst near the boundaries and it does not disappear quickly: at $n = 100$ and $p = 0.1$ the coverage is still 93.2 per cent.

The pathological case shows the mechanism. If $k = 0$ , then $\hat{p} = 0$ , the estimated standard error is zero, and the Wald interval is the single point $[0, 0]$ : the procedure claims certainty from data that plainly does not supply it. The Wilson interval repairs this by not substituting $\hat{p}$ for $p$ in the standard error and instead solving the quadratic inequality $| \hat{p} - p | \leq 1.96 \sqrt{p (1 - p) / n}$ for $p$ . It is

\frac{\hat{p} + \frac{z^{2}}{2 n} \pm z \sqrt{\frac{\hat{p} (1 - \hat{p})}{n} + \frac{z^{2}}{4 n^{2}}}}{1 + \frac{z^{2}}{n}}

which is uglier and behaves far better: at $n = 20$ and $p = 0.1$ its coverage is 95.7 per cent, and with $k = 0$ out of 20 it gives $(0, 0.161)$ instead of a point. That upper limit is close to the familiar rule of three, which says that seeing no events in $n$ trials is consistent with a rate up to about $3 / n$ , here $0.15$ ; the exact one-sided 95 per cent limit is $1 - {0.05}^{1 / 20} = 0.139$ .

Example. In a trial, 0 of 50 patients had a serious adverse reaction. What upper limit on the true rate does that support?

The rule of three gives $3 / 50 = 0.06$ , so rates up to about 6 per cent are consistent with seeing nothing. The exact one-sided 95 per cent limit is $1 - {0.05}^{1 / 50} = 0.0582$ . Reporting "no adverse reactions were observed" without this limit invites the reader to conclude the rate is near zero, when a rate of one in twenty would produce this result about 8 per cent of the time.

Now you. A survey of 400 finds 240 in favour. Give the Wald interval, and say why it is reliable here when it was not in the example above.

Answer

$0.60 \pm 1.96 (0.0245) = (0.552, 0.648)$ . Both $n \hat{p} = 240$ and $n (1 - \hat{p}) = 160$ are far from the boundaries, which is exactly the regime where the normal approximation to the binomial is accurate, so the Wald coverage here is close to its nominal value.

Choosing the sample size

Since the margin is $z σ / \sqrt{n}$ , fixing the margin fixes $n$ . For a proportion in the worst case $p = 0.5$ , a margin of 3 percentage points needs $n = {1.96}^{2} (0.25) / {0.03}^{2} = 1068$ , which is where the standard poll size comes from. A margin of 2 points needs 2401 and a margin of 1 point needs 9604.

This is the $1 / \sqrt{n}$ law from the first lesson, written as a budget. It also makes the honest planning question visible: what width would make the result useful? An interval on a treatment effect running from "harmful" to "wildly beneficial" answers nothing, however correctly it was computed, and the time to discover that is before the study, not after.

Example. A study can afford 1000 participants and expects a proportion near 0.5. What margin can it promise, and what would halving the margin cost?

The margin is $1.96 \sqrt{0.25 / 1000} = 0.031$ , about 3 points. Halving it to 1.55 points requires four times the sample, 4000 participants.

Now you. What sample size gives a margin of 2 percentage points at $p = 0.5$ ?

Answer

$n = {1.96}^{2} (0.25) / {0.02}^{2} = 2401$ .

What the interval does not cover

The width of a confidence interval is computed entirely from sampling variability, so it describes only the error that random sampling produces. It is silent about everything else: a biased frame, differential non-response, an instrument reading high, a question worded to lead. The third lesson's Literary Digest interval would have been about 0.06 points wide and would have missed the truth by 17.8.

It is also silent about the model. A $t$ interval assumes a normal population and independent observations, and a survey that ignores clustering can produce an interval 40 per cent too narrow, as the design effect calculation showed. And an interval is a statement about one parameter: constructing twenty of them and reporting the one that excludes zero destroys the coverage property, a point the lesson on errors takes up directly.

With intervals available, one whole class of question is answered: how large is the effect, with what precision. The other class is whether there is any effect at all, and although the two turn out to be the same question in different clothing, the second has its own machinery, its own vocabulary and its own famous ways of going wrong. That is next.