From probability to inference

Probability answers questions of the form "given this coin is fair, how often will 100 tosses give at least 60 heads", and the answer is a number computed from the model. Statistics asks the question that actually comes up: someone tossed a coin 100 times, got 60 heads, and wants to know whether the coin is fair. This lesson sets up that reversal carefully, because almost every mistake later in the subject is a failure to keep the two directions apart.

The direction of the arrow

In a probability problem the model is given and the data is uncertain. A fair die has a known distribution, and the question is what a batch of rolls will look like. Every quantity in the model, the $p$ of a binomial, the $λ$ of a Poisson, the $μ$ and $σ$ of a normal, is handed over at the start, and the work is deduction: a chain of correct steps from the model to a probability.

In a statistical problem the data is given and the model is uncertain. Someone hands over 100 tosses with 60 heads, or 130 body temperatures, or the yields from 24 plots of barley, and the question is what produced them. Nothing about the mechanism was supplied. The arrow points backwards, from the outcome to the cause, and reversing an arrow is exactly the operation that probability warned about: many models can produce the same data, so the data cannot single one out.

That is not a technicality to be engineered away. It is the permanent situation. A fair coin gives at least 60 heads in 100 tosses about 2.8 per cent of the time, which is uncommon but hardly impossible, so 60 heads is genuinely compatible with fairness and also genuinely compatible with a coin biased to 0.6. No amount of cleverness extracts certainty from that. What statistics does instead is quantify how well each model accounts for what was seen, and attach to every answer a statement about how often that kind of answer is wrong.

The discipline built on that is younger than it looks. Probability was mature by 1800; the machinery of this course was assembled between about 1890 and 1935 by a small number of people working on real problems, Karl Pearson on biological measurement, William Gosset on brewing, Ronald Fisher on agricultural field trials, Jerzy Neyman and Egon Pearson on the logic of decisions. The dates matter because the concepts still carry the shape of the problems they were invented for.

Population, sample, parameter, statistic

Four words do most of the work, and using them loosely is the source of a great deal of confusion.

The population is every unit the question is about. It may be finite and countable, such as the 47.6 million people on the United Kingdom parliamentary electoral register in December 2019, or hypothetical and unbounded, such as every measurement that a given instrument could produce. The sample is the subset actually measured. The parameter is a number describing the population, such as its mean $μ$ or its proportion $p$ , and it is fixed, unknown and usually unknowable. The statistic is a number computed from the sample, such as the sample mean $\overline{x}$ or the sample proportion $\hat{p}$ , and it is known exactly, because it was calculated, and it is different for every sample.

The notation follows the split without exception. Greek letters for parameters, Latin letters or hatted symbols for statistics. When a formula in this course mixes them, that is the point: it is a bridge between the two worlds, and finding those bridges is what the next few lessons do.

The word "average" wrecks this distinction by covering both, which is why it is avoided from here on. The population mean $μ$ and the sample mean $\overline{x}$ are not two names for one thing. One is a fact about the world that no one has access to; the other is an arithmetic result that will change tomorrow if a different sample is drawn.

The estimate is itself a random variable

Here is the pivot on which the whole subject turns. Because the sample was chosen by a random mechanism, any number computed from it is a function of random quantities, which makes it a random variable in exactly the sense of the probability course. It has a distribution, a mean and a variance, and those can be worked out from the model even though the parameter cannot.

Take a die as the population, since its parameters are known and the reasoning can be checked against them. The population mean is $μ = 3.5$ and the population variance is

σ^{2} = \frac{1 + 4 + 9 + 16 + 25 + 36}{6} - {3.5}^{2} = \frac{91}{6} - 12.25 = 2.9167

so $σ = 1.7078$ . Now roll it ten times and record the mean. A real run of ten gave 4, 2, 6, 3, 1, 5, 5, 2, 4, 6, summing to 38, so $\overline{x} = 3.8$ . That is not 3.5, and nothing has gone wrong: the estimate missed by 0.3 for no reason other than which faces came up.

Roll another ten and $\overline{x}$ will be something else. The set of values $\overline{x}$ takes across all possible samples, with their probabilities, is called the sampling distribution of $\overline{x}$ , and it is the single most important object in the subject. It exists whether or not anyone computes it, and it is what makes a statement about one sample into a statement about the world.

Example. For the mean of ten rolls, what are the mean and standard deviation of the sampling distribution?

The rolls are independent with mean 3.5 and variance 2.9167 each. Expectation is linear, so $E [\overline{X}] = 3.5$ exactly. Variances of independent variables add, and dividing by ten divides the variance by $10^{2}$ , so $Var (\overline{X}) = 2.9167 / 10 = 0.29167$ and the standard deviation is $\sqrt{0.29167} = 0.5401$ .

Now you. What are the mean and standard deviation of the sampling distribution of $\overline{X}$ for forty rolls?

Answer

The mean is still 3.5. The standard deviation is $\sqrt{2.9167 / 40} = 0.2700$ , exactly half the value for ten rolls, because forty is four times ten and the square root of four is two.

How far off is one sample likely to be

That last calculation is already an answer to the practical question, and it is worth pressing on it. The sampling distribution of the mean of ten rolls can be computed exactly, by convolving the die distribution with itself ten times, and it gives

P (3.0 \leq \overline{X} \leq 4.0) = 0.6870

so a single sample of ten misses the truth by more than half a point about 31 per cent of the time. For forty rolls the same calculation gives 0.9426, and for the estimate to be reliably close, the sample has to be large.

Notice the shape of the improvement. Quadrupling the sample from ten to forty halved the standard deviation, because the standard deviation of the mean is $σ / \sqrt{n}$ and the square root is doing the damage. Halving an error costs four times the data; cutting it to a tenth costs a hundred times. That single fact governs the cost of every survey, trial and experiment in this course, and it is the reason large studies are rare.

It also explains something that sounds wrong when first heard. A national poll of 1000 people reports a margin of error of about 3 percentage points, and the same 1000 people give the same 3 points whether the country has 5 million voters or 300 million. The finite population correction that accounts for the population's size is $\sqrt{(N - n) / (N - 1)}$ , which for $N = 60$ million and $n = 1000$ equals 0.99999, a correction in the fifth decimal place. What matters is the size of the sample, not the fraction of the population it represents.

Example. In a survey of 1000 people, 520 say they will vote yes. The standard deviation of $\hat{p}$ is $\sqrt{p (1 - p) / n}$ . Estimate it, and give the usual two-standard-error range.

Using $\hat{p} = 0.52$ in place of the unknown $p$ , the standard error is $\sqrt{0.52 \times 0.48 / 1000} = 0.0158$ . Multiplying by 1.96 gives 0.031, so the familiar "plus or minus 3 points". The later lessons make the logic of that interval precise; the arithmetic is already available.

Now you. How large a sample is needed to bring $1.96 \sqrt{p (1 - p) / n}$ down to 1 percentage point, in the worst case $p = 0.5$ ?

Answer

Set $1.96 \sqrt{0.25 / n} = 0.01$ , so $n = {1.96}^{2} \times 0.25 / 0.0001 = 9604$ . Ten times the precision of a 96-person survey costs a hundred times the people.

Two kinds of error, and only one of them shrinks

The die example had one source of error: which faces happened to come up. That is sampling error, it is unavoidable, it is quantifiable, and it shrinks as $1 / \sqrt{n}$ . Almost all of the mathematics in this course is about sampling error, for the simple reason that it is the part that can be described by a formula.

The other kind is bias, meaning any mechanism that pushes the estimate away from the truth in a consistent direction. A thermometer reading 0.4 degrees high, a survey that reaches only people who answer the telephone at two in the afternoon, a study that measures only the patients who came back for a follow-up. Bias does not shrink with $n$ . Collecting ten thousand readings from the same miscalibrated thermometer produces a very precise estimate of the wrong number, and the standard error shown alongside it, being computed from the spread of the readings, will look reassuringly small.

That asymmetry is the most practically important sentence in this lesson. The quantity a statistical formula reports is the sampling error, and it says nothing whatever about the bias. A published figure of "43 per cent, plus or minus 3 points" is a claim about the arithmetic of random sampling and not a claim that the true value is near 43. It is near 43 only if the sample was drawn in a way that gives everyone a known chance of appearing, which is the subject of the sampling lesson later on and the reason that lesson exists.

Example. A bathroom scale reads 0.8 kg heavy. A person weighs themselves 100 times, and the readings vary by about 0.3 kg from the day-to-day noise. What does the standard error of the mean reading say about the accuracy of the result?

The standard error is $0.3 / \sqrt{100} = 0.03$ kg, so the mean of the readings pins down what the scale says to within a few tens of grams. What the scale says is 0.8 kg away from the truth, and every one of the 100 readings is 0.8 kg off in the same direction, so the reported precision describes the noise and completely misses the error.

Now you. The person takes 10000 readings instead. What is the standard error, and how far is the mean from the true weight?

Answer

The standard error falls to $0.3 / \sqrt{10000} = 0.003$ kg, and the mean is still 0.8 kg too high. More data buys precision and buys nothing at all against bias.

What a statistical answer is allowed to claim

Given that the arrow cannot be reversed with certainty, what exactly is being sold? Three kinds of claim, and each is more modest than the sentence people usually take away from it.

An estimate is a single number offered as the best available guess of a parameter, together with a standard error saying how much it would move if the study were repeated. A confidence interval is a range constructed by a procedure that captures the true parameter in a stated proportion of repetitions, 95 per cent by convention. A test reports how unusual the data would be if a specified model were true. All three are statements about the behaviour of a procedure over repeated use, not statements about the certainty of one particular conclusion.

That is a genuine limitation and it is worth stating plainly rather than glossing. Statistics does not tell you whether this coin is fair. It tells you that a rule you might adopt for deciding about coins gets it wrong at most five per cent of the time when the coin is in fact fair, and you get to choose the rule. A reader who wants a probability attached to the hypothesis itself is asking a Bayesian question, and the probability course showed that answering it requires a prior. The frequentist machinery developed here avoids the prior by changing the question, and knowing exactly which question is being answered is what separates using these tools from being used by them.

What is coming

The plan follows directly from this lesson. A sample has to be reduced to a few numbers before anything can be said about it, and the choice of those numbers is not innocent, so that is next. Then comes the collection itself, since a statistic computed from a bad sample is arithmetic without meaning. After that the sampling distribution can be derived properly rather than sketched, which unlocks estimation, intervals, tests, comparisons and the fitting of lines.

The last lessons return to the point made in passing above. Everything in the middle of the course is correct, and correct is not sufficient, because the interesting failures happen at the joins: in how the data was gathered, in which analysis was chosen after seeing it, and in what the resulting number was made to say.