Where the data comes from

Every formula in the rest of this course begins with the phrase "let $X_{1}, dots, X_{n}$ be independent draws from the population", and that phrase is an assumption about how the data was collected rather than a property of the numbers. This lesson is about earning it, and about the two ways studies routinely fail to.

The frame, and the gap in front of it

Three things are easy to run together. The target population is who the question is about: adults in France, patients with a particular diagnosis, screws produced by a machine this week. The sampling frame is the list actually available to sample from: a register, a customer database, a set of telephone numbers, the crate on the bench. The sample is what comes back.

The frame is never the population. A telephone frame omits people without telephones and includes numbers that no longer belong to anyone. A patient register omits those who never sought care. The difference is coverage error, and it is a property of the list, so it is fixed before a single observation is taken and cannot be detected from the data afterwards. A perfectly executed random sample from the wrong frame is a perfectly executed measurement of the wrong population.

Naming the frame explicitly is therefore the first question to ask about any statistic, and it is the question most often skipped. When a study reports "a nationally representative sample of 2000 adults", the frame is usually a panel of people who agreed at some earlier point to answer surveys, which is a real population but not the nation.

Simple random sampling

The base case is the simple random sample: every subset of $n$ units from the frame is equally likely to be chosen. The important consequence is not fairness in a moral sense but that each unit's chance of inclusion is known, which is exactly what licenses treating $X_{1}, dots, X_{n}$ as draws from the population distribution and applying everything the probability course proved about independent variables.

Selecting one requires a random mechanism and nothing else: number the frame and draw numbers. What it emphatically does not mean is choosing "without a system", because people asked to pick arbitrarily produce patterns. Studies of haphazard selection show consistent preferences for the middle of a range and against repeats, which is why sampling is done with a generator and a seed, recorded, so the selection can be audited.

Simple random sampling has one practical defect: it takes no account of structure that is already known. If half the population is known to differ systematically from the other half, an SRS will by chance include an unrepresentative mix of the two, and that chance variation is pure added noise. The next design removes it.

Stratified sampling

Split the frame into strata that are internally similar, then sample independently within each. If stratum $h$ holds a fraction $W_{h}$ of the population and has variance $σ_{h}^{2}$ , and $n_{h}$ units are drawn from it, the estimator ${\overline{x}}_{s t} = \sum_{h} W_{h} {\overline{x}}_{h}$ has variance

Var ({\overline{x}}_{s t}) = \sum_{h} W_{h}^{2} \frac{σ_{h}^{2}}{n_{h}}

by independent additivity across strata. Compare that with simple random sampling, whose variance is $σ^{2} / n$ where $σ^{2}$ is the whole population variance. The population variance splits into a within-stratum part and a between-stratum part, $σ^{2} = \sum_{h} W_{h} σ_{h}^{2} + \sum_{h} W_{h} (μ_{h} - μ)^{2}$ , and allocating $n_{h} = W_{h} n$ makes the stratified variance equal to $\frac{1}{n} \sum_{h} W_{h} σ_{h}^{2}$ , the within part alone. Stratification deletes the between-stratum variance from the answer, and the more the strata differ from each other, the more it deletes.

Example. A town of 10,000 households has 8000 in flats, spending a mean of 20 with standard deviation 5, and 2000 in houses, spending a mean of 60 with standard deviation 10. With a total sample of 100, compare simple random sampling against stratified sampling with 80 and 20.

The population mean is $0.8 (20) + 0.2 (60) = 28$ . The within part is $0.8 (25) + 0.2 (100) = 40$ and the between part is $0.8 (20 - 28)^{2} + 0.2 (60 - 28)^{2} = 51.2 + 204.8 = 256$ , so $σ^{2} = 296$ . Simple random sampling gives variance $296 / 100 = 2.96$ and standard error 1.72. Stratified gives $0.64 (25) / 80 + 0.04 (100) / 20 = 0.2 + 0.2 = 0.4$ and standard error 0.63. The variance fell by a factor of 7.4, from a design change that cost nothing.

Now you. Optimal allocation puts $n_{h}$ in proportion to $W_{h} σ_{h}$ rather than $W_{h}$ . Find the allocation and the resulting variance.

Answer

The products are $0.8 (5) = 4$ and $0.2 (10) = 2$ , in the ratio 2 to 1, so 67 flats and 33 houses. The variance becomes $(\sum_{h} W_{h} σ_{h})^{2} / n = 36 / 100 = 0.36$ , a further ten per cent below proportional allocation, because the more variable stratum gets more of the sample.

Clusters, and the price of convenience

Sometimes reaching a random individual is expensive and reaching a whole group is cheap. Cluster sampling selects groups, schools, villages, hospital wards, and measures everyone inside the chosen ones. It saves an enormous amount of travel and administration, and it costs precision, because people in the same cluster resemble each other, so the second person from a village carries less new information than a fresh village would.

The cost is measured by the design effect, the ratio of the true variance of the estimate under the design to what it would have been under simple random sampling of the same size. For cluster sampling with clusters of size $m$ and intraclass correlation $ρ$ , the design effect is approximately $1 + (m - 1) ρ$ . With clusters of 20 and a modest $ρ = 0.05$ , that is $1 + 19 (0.05) = 1.95$ , so 2000 people sampled in clusters carry about the information of 1025 sampled individually. The effective sample size is the actual size divided by the design effect, and it is the number that should be used in every formula later in this course. Software that is told only the sample size will report standard errors roughly $\sqrt{1.95} = 1.4$ times too small.

Real surveys combine all of this: stratify by region, sample areas within region, sample households within area, and weight the result back to known population totals. The weights are what make the estimator unbiased for the population rather than the sample, and a weighted survey analysed as though it were a simple random sample is one of the most common technical errors in published work.

The two failures no sample size fixes

In 1936 the Literary Digest mailed about ten million ballots for the United States presidential election and received 2,376,523 back. That is the largest election survey ever conducted, and it predicted Alf Landon over Franklin Roosevelt by 57 per cent to 43. Roosevelt won 60.8 per cent of the popular vote and 523 of 531 electoral votes. George Gallup, using around 50,000 people chosen to match the demographics of the electorate, called the result correctly.

It is worth putting the magnitude of that failure in the units of the last lesson. With $n = 2 {,} 376 {,} 523$ , the standard error of a proportion near a half is $\sqrt{0.25 / n} = 0.00032$ , which is 0.032 percentage points. The Digest's error was 17.8 points, about 550 standard errors. There is no sample size at which that becomes plausible as sampling error, so it was not sampling error.

Two mechanisms produced it, and both are permanent hazards. The frame was built from telephone directories, automobile registrations and the magazine's own subscriber list, which in 1936 selected for wealth, and wealth correlated strongly with voting Republican: that is coverage error. And only 24 per cent of the ballots came back, with Landon's supporters, who were more motivated against an incumbent, returning them at a higher rate: that is non-response bias.

Non-response is the more dangerous of the two now, because coverage has improved and response rates have collapsed. The Pew Research Center's telephone surveys had a response rate of 36 per cent in 1997 and 6 per cent in 2018. A 6 per cent response rate does not automatically mean a biased estimate, since it depends entirely on whether responding is related to the answer, but it means nothing in the data can rule the bias out, and the survey's own margin of error describes only the 6 per cent.

Example. A survey reaches 10,000 people, of whom 3000 respond, and 60 per cent of respondents support a proposal. Unknown to the researchers, only 40 per cent of the non-respondents support it. What is the true population figure, and how does the bias compare with the reported standard error?

The truth is $0.3 (60) + 0.7 (40) = 46$ per cent, so the estimate of 60 is 14 points too high. The reported standard error is $\sqrt{0.6 \times 0.4 / 3000} = 0.0089$ , or 0.89 points, so the bias is about 16 standard errors. The published interval, 60 plus or minus 1.8 points, does not come close to containing the truth.

Now you. With a 20 per cent response rate, 55 per cent support among respondents and 35 per cent among non-respondents, find the true figure and the bias.

Answer

$0.2 (55) + 0.8 (35) = 39$ per cent, so the estimate is 16 points too high. A lower response rate gives the unobserved majority more weight, which is why response rate matters even though it appears in no formula.

Samples nobody designed

Much of the data analysed today was not sampled at all. Transaction logs, hospital records, social media posts and scraped web pages are found data: whatever happened to be recorded. The temptation is that they are enormous, and the previous lesson's warning applies at full force, since size is exactly the dimension along which found data is impressive and exactly the one that does not help.

There is a clean way to state the trade. For estimating a population mean from a sample of size $n$ out of a population of $N$ , the error decomposes into a term controlled by the sample fraction and a term measuring the correlation between being in the sample and the value being measured. When selection is random that correlation is zero and only the first term survives, giving the familiar $1 / \sqrt{n}$ . When selection is not random, the correlation term does not shrink with $n$ at all, and it is multiplied by $\sqrt{N / n}$ , which for a large found dataset from a much larger population is a big number. A correlation of 0.001 between responding and the answer can wipe out the advantage of millions of records.

That is the formal version of a plain point: a badly chosen sample of a million is worth less than a well chosen sample of a thousand, and it is worse than useless if its size persuades the reader to trust it.

Example. A shop's loyalty card database records the purchases of 400,000 customers, and management wants the mean spend per visit across all customers. What is the frame, and what is the likely direction of the coverage error?

The frame is holders of a loyalty card who used it, which excludes customers who never signed up and visits where the card was forgotten. Card holders are the more regular customers, and small quick visits are the ones where a card is least likely to be presented, so the mean spend from this frame is biased upwards. Nothing in the 400,000 records reveals the size of that bias, since the excluded visits are absent by construction.

Now you. Name a cheap study that would measure the size of the bias.

Answer

Take a random sample of visits at the till, whether or not a card is presented, and compare the mean spend of the carded visits with the mean over all sampled visits. A small properly drawn sample calibrates the large improper one, which is the standard repair.

What the formulas will assume

From the next lesson onwards, the working assumption is a simple random sample from a population with mean $μ$ and variance $σ^{2}$ , and everything derived rests on it. Independence between observations gives the addition of variances. Identical distribution gives a single $μ$ and $σ$ to estimate. Random selection is what makes the sample mean centred on the population mean rather than on something else.

When a design departs from that, and real designs always do, the repairs are known: weights for unequal selection probabilities, effective sample size for clustering, stratum-level formulas for stratification. What has no repair is a frame that excludes the people the question is about, or a response mechanism related to the answer. Those are decided before the analysis begins, and the honest report says what the frame was and what the response rate was, so a reader can judge them.

With the collection settled, the sample mean can be treated as a random variable and its distribution worked out exactly, which is what makes any of the numbers computed in the previous lesson into evidence. That derivation is next.