Nothing in the previous thirteen lessons is wrong. Applied once, to a pre-specified question, on data collected as described, each method delivers exactly what it advertises. The trouble is that published statistics are almost never produced that way, and the gap between the idealised procedure and the real one is where most false findings live. This last lesson names the mechanisms, each next to the tool that already handles it.
Correct arithmetic, false conclusion
Start with the arithmetic of the errors lesson, because it frames everything else. In a field where one hypothesis in ten is true, running studies at 5 per cent significance with 80 per cent power, about a third of the significant findings are false. That is the best case: correct methods, honest researchers, adequate power.
Now degrade any single input. Drop the power to 20 per cent, which was the median estimate for neuroscience in 2013, and the majority of significant findings are false. Let researchers try several analyses and report the best, and the effective rises far above 0.05. Let journals publish only significant results, and the false ones are preferentially retained while the true nulls vanish.
The empirical check on all this came in 2015, when the Open Science Collaboration repeated 100 psychology studies with larger samples and pre-registered protocols. Of the 97 original studies that had reported significant effects, 35 of the replications were significant, and the replication effect sizes averaged about half the originals. Both numbers are what the mechanisms below predict, and the halving is the exaggeration ratio of the power lesson.
The multiplicity nobody counts
Testing twenty independent hypotheses at the 5 per cent level gives a chance of at least one significant result when every null is true. That much is familiar, and the repairs are standard: the Bonferroni correction tests each at , so 0.0025 for twenty, guaranteeing the overall error rate stays at 5 per cent at the cost of power; the Benjamini-Hochberg procedure instead controls the false discovery rate, the expected fraction of the rejections that are false, which is less severe and usually more appropriate when many hypotheses are genuinely worth testing.
The harder version is the multiplicity that never appears in a table. A researcher with one dataset makes dozens of defensible choices: which outliers to exclude, whether to log-transform, which covariates to adjust for, where to cut a continuous variable into groups, which of several related measures is the outcome. Each is defensible in isolation, and each was chosen after seeing the data. Andrew Gelman named this the garden of forking paths, and its distinctive feature is that no multiple testing correction can repair it, because the researcher performed exactly one test and honestly reports one -value. The other tests were never run; they were merely available.
Simmons, Nelson and Simonsohn demonstrated the scale of this in 2011 by simulating four ordinary flexibilities in combination, and found the false positive rate for a null effect rising from 5 per cent to 61 per cent. They also ran a real experiment using the same freedoms and obtained significant evidence that listening to a particular song made subjects a year and a half younger, which is the point made with maximum force.
The fix is not statistical. It is pre-registration: writing down the outcome, the analysis and the exclusion rules before the data exists, so the number of tests is fixed in advance and anything else is labelled exploratory. Exploratory analysis is legitimate and valuable; presenting it as confirmatory is not.
Peeking
A special and very common forking path is looking at the data as it accumulates and stopping when the result is significant. It feels efficient, and it destroys the error rate.
The size of the damage is easy to simulate. Take a genuinely null effect, test at each of several equally spaced interim points, and stop at the first significant result. With one look the type I error is the nominal 5 per cent. With two it is 8.2 per cent, with three 10.7, with five 14.2, with ten 19.4, with twenty 24.6 and with fifty 32.2. Testing after every single observation, with no limit on sample size, reaches significance with probability 1.
The repair is well developed, because clinical trials genuinely need to stop early when a treatment is working or harming. Group sequential designs spend the error budget across a pre-specified number of looks, using boundaries such as O'Brien-Fleming's that require very strong evidence early and approach the ordinary threshold at the end. The essential feature is that the number and timing of the looks is fixed in advance, which is what makes the total error rate computable.
Example. A researcher tests at five interim points and stops at the first . What is the true type I error rate, and what would a Bonferroni-style correction require instead?
The simulated rate is 14.2 per cent, nearly three times nominal. Dividing the threshold by 5 gives 0.01 at each look, which is conservative for this purpose since the looks are highly correlated, but it is a safe crude repair.
Now you. With ten looks the rate is 19.4 per cent. Roughly how many false positives per hundred null studies does that habit add?
Answer
About 14 extra per hundred, from 5 to 19. Every one of them is reported as a -value below 0.05 with no indication that anything unusual was done.
What never got published
Suppose twenty independent groups test the same null hypothesis, which is true. One gets by chance. If the nineteen null results are unpublishable, and the one positive appears in a journal, the literature contains a single significant finding and no trace of the nineteen that contradicted it.
Publication bias is that mechanism operating continuously. Its usual diagnostic is the funnel plot: effect size on one axis against precision on the other, which should form a symmetric funnel, wide at the imprecise end and narrowing towards the true value. Missing small studies with small or negative effects show up as a bitten-off corner. The asymmetry is suggestive rather than conclusive, since small studies can genuinely differ from large ones in how they are conducted.
The consequences are worst where they matter most. The main institutional repair is trial registration, mandatory for clinical trials since around 2005, which creates a public record of every study that started, so a study that never reports can be counted. Comparisons of registered trials against their publications routinely find outcomes that were measured and never reported.
Relative risk without a base rate
A treatment reduces the risk of an event from 2 in 1000 to 1 in 1000 per year. Reported as a relative risk, that is a 50 per cent reduction. Reported as an absolute risk, it is 1 case per 1000 people per year, so a thousand people must be treated for a year to prevent one event. Both descriptions are exactly true and they support entirely different decisions, and the relative one is chosen for headlines with striking consistency.
In October 1995 the UK Committee on Safety of Medicines warned that third-generation oral contraceptives roughly doubled the risk of venous thromboembolism. The doubling was real. In absolute terms the risk went from about 15 to about 25 cases per 100,000 women per year, an excess of 1 case per 10,000 women per year, against a risk from pregnancy itself that is considerably higher. Large numbers of women stopped taking the pill, and the following year saw a marked rise in unintended pregnancies and abortions in England and Wales. The statistic was correct and the framing did the harm.
The rule that follows is short: never report a relative change without the base rate it applies to, and when reading one, ask for the base rate before forming any view. The same applies to the reverse trick of quoting an absolute risk to make a real hazard sound negligible.
Example. A study reports that a food is associated with a 20 per cent increase in the risk of a cancer whose lifetime incidence is 6 in 100. What is the absolute change?
A 20 per cent increase on 6 per cent is 7.2 per cent, so 1.2 additional cases per 100 people over a lifetime. That is a real effect and it is nothing like the impression "20 per cent more cancer" creates.
Now you. The same 20 per cent relative increase applies to a cancer with a lifetime incidence of 1 in 10,000. What is the absolute change?
Answer
From 1 to 1.2 per 10,000, an excess of 2 cases per 100,000 people over a lifetime. The identical relative figure describes an effect a thousand times smaller.
Outcomes swapped and subgroups mined
Two related practices are worth naming because they are easy to detect in a published paper.
Outcome switching is measuring several outcomes, then reporting whichever moved as though it had been the primary one. Registration makes this visible, since the registered primary outcome is on record. A paper whose headline result is not its registered primary outcome should be read as exploratory whatever it says.
Subgroup analysis splits the sample after the fact and reports the split where the effect appears. With enough subgroups something always appears, and the classic demonstration is in the ISIS-2 trial of aspirin after heart attack, whose authors deliberately included a subgroup analysis by astrological birth sign: aspirin appeared not to work for Geminis and Libras, and worked strongly for everyone else. The overall effect was large and real, and the subgroup finding was noise, presented precisely to teach the reader not to trust subgroup findings.
A pre-specified subgroup, stated before the data and with an interaction test rather than two separate tests, is legitimate. The number of subgroups examined is the thing that must be reported, and it almost never is.
Example. A trial with a null treatment effect examines 12 subgroups, testing each at the 5 per cent level. What is the chance of at least one significant subgroup?
, so almost half of such trials will produce a subgroup finding to headline. If the paper reports one significant subgroup out of twelve examined, that is exactly what a null effect looks like.
Now you. How many subgroups give a better than even chance of at least one significant result?
Answer
Solve , so , which means 14 subgroups.
Pictures that do the work
Charts mislead through a small and repeatable set of devices, and knowing them makes the check quick.
A truncated vertical axis magnifies a small difference into a visual chasm, and it is legitimate only when the reader is told and the quantity has no meaningful zero. Bar charts are the worst case, since a bar's length is the encoding and a truncated bar encodes nothing. A dual axis chart with two differently scaled vertical axes can be made to show almost any relationship between two series by choosing the scales, and should be read as decoration rather than evidence.
Encoding a one-dimensional quantity as the area or volume of a shape multiplies the apparent change: doubling the radius of a circle quadruples its area. Plotting cumulative totals turns any positive quantity into a smooth rising curve, so a cumulative chart cannot show a decline and should not be used to argue there was none. And a scatter with a fitted line and no scatter shown, or an average of averages that ignores group sizes, hides exactly the structure the twelfth and thirteenth lessons were about.
The questions that catch most of it
A short checklist, each item tied to the lesson that supplies it.
What is the population, and what was the frame? Non-response and coverage are not in any interval. What was the sample size, and what effect would it have had a reasonable chance of detecting? An underpowered study's significant result is inflated by a computable factor. How many things were tested, including the analyses that were tried and discarded? Was the outcome and the analysis specified before the data? Is there a control group, and was assignment randomised, and if not, what confounder is being assumed away? Is the effect reported in absolute terms with its base rate? Is the interval reported, or only the verdict? And who would have published this if it had come out the other way?
None of these questions requires arithmetic. All of them require the arithmetic of this course to know why they are the right questions, which is the difference between distrusting statistics and being able to use them. The methods are not the weak point. The joins are: how the data was made, what else was tried, and what was left out of the sentence at the top of the article.