Every technique so far establishes that two things are associated: a difference between groups, a slope, a table that departs from independence. None of them establishes that one causes the other, and this lesson is about the gap. It is the difference between a study that can change a decision and one that cannot.
Four explanations for any association
When and move together in a sample, four explanations are always on the table, and the data alone cannot distinguish them.
causes , which is what the researcher hopes. causes , which is reverse causation and is common wherever the outcome influences who gets the treatment: sicker patients get the aggressive therapy, so the aggressive therapy looks harmful. A third variable causes both, which is confounding. Or the association is an artefact of how the sample was assembled, which is selection.
A confounder is a variable that affects the outcome and is also associated with the exposure, and is not on the causal path between them. That last clause matters: if smoking causes tar deposits which cause cancer, adjusting for tar deposits would remove the very effect being measured. Deciding what is a confounder is a question about the causal structure of the world, not about the data, and it cannot be answered by any test.
The size of the resulting distortion has no bound. It can shrink a real effect, inflate one, or reverse the sign, which is the case worth seeing in real numbers.
Berkeley, 1973
In autumn 1973 the University of California, Berkeley admitted 44 per cent of its 8442 male graduate applicants and 35 per cent of its 4321 female applicants. A nine-point gap on twelve thousand applications is not sampling noise, and the university asked its statisticians to look at it. Bickel, Hammel and O'Connell published the result in Science in 1975.
Split by department, the gap moves the other way. For the six largest departments the figures are
| Department | Men applied | Men admitted | Women applied | Women admitted |
|---|---|---|---|---|
| A | 825 | 62% | 108 | 82% |
| B | 560 | 63% | 25 | 68% |
| C | 325 | 37% | 593 | 34% |
| D | 417 | 33% | 375 | 35% |
| E | 191 | 28% | 393 | 24% |
| F | 373 | 6% | 341 | 7% |
Women are admitted at a higher rate in four of the six, and where they are lower the gaps are 3, 3 and 4 points. Yet aggregated over these six departments, men are admitted at 44.5 per cent and women at 30.3 per cent, a gap of 14 points. Every department is roughly even or favours women, and the total strongly favours men.
The mechanism is visible in the table. Departments A and B admit around 63 per cent of applicants and are overwhelmingly applied to by men: A is 12 per cent female, B is 4 per cent. Departments C, E and F admit between 6 and 35 per cent and receive far more women. Department is a confounder: it determines the admission rate, and it is strongly associated with sex. Aggregating over it compares mostly-men-in-easy-departments with mostly-women-in-hard-departments, and reports the difference as though it were about sex.
A reversal like this is called Simpson's paradox, after a 1951 paper by Edward Simpson, and it is not a paradox at all: it is the ordinary arithmetic of weighted averages, in which the weights can dominate the values being averaged.
Example. Two departments. The first admits 90 per cent of 100 male applicants and 95 per cent of 20 female ones. The second admits 20 per cent of 20 men and 25 per cent of 100 women. Compute the overall rates.
Men: per cent. Women: per cent. Women are admitted at a higher rate in both departments and at less than half the overall rate, because 83 per cent of the men applied to the easy department and 83 per cent of the women applied to the hard one.
Now you. In that example, what would the women's overall rate be if they had applied in the same proportions as the men?
Answer
Apply the women's departmental rates to the men's application counts: per cent, above the men's 78.3. Holding the application mix fixed reverses the comparison back.
Adjustment, and what it buys
The repair used just now has a name: direct standardisation, in which each group's stratum-specific rates are applied to one common population of strata. It removes the confounding by department because it compares like with like.
Applied to the Berkeley six with the combined applicant pool as the standard, the standardised admission rates are 38.8 per cent for men and 42.9 per cent for women. The direction has reversed, and the honest summary of the whole exercise is that the aggregate gap was produced by which departments each sex applied to, which is a real phenomenon with its own causes and is a different claim from the one the aggregate appeared to make.
Multiple regression does the same job in a more general setting. Including a confounder as an extra predictor estimates the effect of the exposure while holding the confounder fixed, so the coefficient on treatment answers "among applicants to the same department, how does sex relate to admission". Adding a term for department to a regression on the Berkeley data does the same work as standardising the tables.
Example. A study finds that coffee drinkers have higher rates of lung cancer. Smoking is more common among coffee drinkers and causes lung cancer. What does stratifying by smoking status do?
Within smokers and within non-smokers separately, coffee drinking and lung cancer can be compared without smoking varying between the groups. If the association disappears in both strata, it was confounded by smoking. If it persists in both, smoking does not explain it, though some other unmeasured variable still might.
Now you. Why would adjusting for "years of tar exposure in the lungs" be a mistake if the question is whether smoking causes cancer?
Answer
Tar exposure is on the causal path from smoking to cancer, not a confounder. Holding it fixed removes the mechanism by which smoking acts, and the adjusted coefficient would understate or erase a real effect.
What adjustment cannot do
Adjustment handles confounders that were measured, measured accurately, and correctly identified as confounders. Each of those three is a real limitation.
Unmeasured confounders are untouched, and there is no way to detect them in the data: a regression with a good and clean residuals is entirely consistent with a large omitted variable. Confounders measured with error are only partly adjusted for, so residual confounding survives in proportion to the measurement error. And a variable that is not a confounder but a consequence of the exposure, or a common effect of exposure and outcome, makes things worse when adjusted for rather than better.
The most expensive demonstration of this is hormone replacement therapy. Large observational studies through the 1980s and 1990s, including the Nurses' Health Study, found that women taking HRT had substantially lower rates of coronary heart disease, on the order of 40 per cent lower, and the analyses adjusted for the confounders anyone could name. The Women's Health Initiative randomised over 16,000 women and reported in 2002 a hazard ratio for coronary heart disease of 1.29, an increase. The observational studies were not sloppy; women who took HRT differed from women who did not in ways nobody had measured, including how much health care they sought and how healthy they were to begin with.
Why randomising works
Assign the exposure by a random mechanism, and the entire problem changes character. Because assignment is independent of everything about the unit, the treated and control groups are drawn from the same population with respect to every variable at once: age, severity, income, genotype, and every variable nobody thought of or could measure. There is no confounding, not because it was adjusted away but because the assignment mechanism made it impossible.
That is the single most important idea in experimental design, and it is worth stating what it does not promise. Randomisation does not guarantee balance in any particular trial. With 100 per arm and a binary covariate at 50 per cent prevalence, the difference in prevalence between arms has a standard error of 7.1 points, so imbalances of 14 points happen one time in twenty. What randomisation guarantees is that the imbalance is random, which is exactly the condition under which the standard errors and -values of the earlier lessons are correct. The uncertainty from imbalance is inside the interval already.
This also explains why testing baseline covariates for significant imbalance is a mistake that appears in a great many published trials. In a properly randomised trial any imbalance is by construction due to chance, so the test has a known and uninteresting answer, and with twenty covariates tested at the 5 per cent level the chance that at least one comes out significant is .
The mechanism has to actually be random. The 1970 United States draft lottery placed 366 capsules in a container month by month and mixed them inadequately, and the resulting draft numbers correlate with birth date at about : men born late in the year were drafted earlier. A procedure that looks random on television is not thereby random, which is why serious trials use generated sequences with concealed allocation.
Example. A trial randomises 50 to each arm. Ages have standard deviation 12 years. How large a difference in mean age between the arms is unsurprising?
The standard error of the difference is years, so differences up to years occur one time in twenty by chance alone.
Now you. With 500 per arm, what is the corresponding figure?
Answer
years. Randomisation balances better in larger trials, at the usual rate, which is one more reason size matters.
The rest of the design
Randomisation removes confounding at assignment, and three further devices deal with what happens afterwards.
Blinding stops knowledge of the assignment from affecting the outcome. Patients who know they received the active drug report improvement differently, and clinicians who know it assess outcomes differently and may treat differently. Double blinding, where neither party knows, is what the Salk trial's placebo-control portion used, and its designers insisted on it precisely because a parallel non-randomised portion of the same trial existed for comparison and was much harder to interpret.
Blocking reduces variance rather than bias. If the units come in naturally similar groups, plots in the same field, litters of the same mother, randomise within each block. This is the paired design of the two-group lesson generalised, and it removes block-to-block variation from the comparison exactly as stratification removed stratum-to-stratum variation from a survey.
Stratified randomisation, in which assignment is randomised separately within levels of a strong prognostic variable, guarantees balance on that variable instead of leaving it to chance. It is standard in multi-centre trials, where centre is the stratifying variable.
Fisher assembled all of this at Rothamsted in the 1920s, on agricultural field trials, and the reason the argument is airtight is that it does not depend on knowing which variables matter. That is what makes it work in medicine and in fields where the important variables are unknown.
When randomising is impossible
A great deal cannot be randomised: nobody assigns people to smoke, to be poor, or to live near a motorway. Those questions still have answers, and the standards for them are correspondingly higher.
The classical example is smoking and lung cancer. Doll and Hill's 1950 case-control study and their subsequent cohort of British doctors were observational, and Fisher argued publicly through the 1950s that a genetic predisposition might cause both the smoking and the cancer, which is a coherent confounding hypothesis that no single study could rule out. What settled it was a set of features no single association has: a large effect, a dose-response relationship, the right temporal order, consistency across many populations and designs, a plausible mechanism, and the fall in risk after quitting. Austin Bradford Hill set these out in 1965, and they are viewpoints for judgement rather than a checklist to be scored.
Modern practice adds designs that borrow some of randomisation's logic: instrumental variables, which use a source of variation in the exposure that is plausibly unrelated to the outcome; regression discontinuity, which compares units just either side of an arbitrary threshold; and difference in differences, which uses a change in policy affecting one group and not another. Each rests on an assumption that cannot be verified from the data, which is the permanent price of not being able to randomise.
The tools are now complete. What remains is the gap between using them correctly on one question and the way results actually reach print, which is the last lesson.