Testing a hypothesis

A confidence interval answers "how large is it, and how precisely do we know". A great many questions are cruder than that: does this drug do anything, is this coin bent, did the yield actually change. Testing is the machinery for those, and it is simultaneously the most used and the most misdescribed tool in the subject.

The question a test answers

The logic is a proof by contradiction with the certainty removed. To argue that an effect exists, assume it does not, work out what the data would then look like, and see whether what was observed is the kind of thing that assumption comfortably produces. If it is not, the assumption is doubted. If it is, nothing has been shown either way.

The assumption is the null hypothesis $H_{0}$ , and it must be specific enough to compute with: $p = 0.5$ , or $μ = 98.6$ , or "the treatment and control distributions are identical". The alternative $H_{1}$ is what is entertained instead, usually a whole range such as $p \neq 0.5$ . The asymmetry is deliberate and permanent: the null is the one whose consequences can be calculated, so it is the one that gets tested, and it is never the one that gets proved.

That asymmetry is the source of the most common error in reporting. Failing to reject the null is not evidence that the null is true. A study of eight patients will fail to reject almost any null, because it has no ability to detect anything, and "no significant difference was found" from such a study means only that the study was incapable of finding one. The next lesson makes that precise.

The test statistic and the $p$ -value

A test statistic is a number computed from the data whose distribution under $H_{0}$ is known, and which is large when the data disagrees with $H_{0}$ . For a mean it is usually

t = \frac{\overline{x} - μ_{0}}{s / \sqrt{n}}

the distance from the null value measured in estimated standard errors, which under $H_{0}$ follows the $t$ distribution of the previous lesson.

The $p$ -value is the probability, computed under $H_{0}$ , of getting a test statistic at least as extreme as the one observed. That is the whole definition, and every word in it is load-bearing. It is a probability about data, not about hypotheses. It is conditional on the null being true. And "at least as extreme" means the tail, not the point, since for continuous data any exact value has probability zero.

A small $p$ -value says the observed data would be unusual if $H_{0}$ held. That is grounds for doubting $H_{0}$ , and it is not a probability that $H_{0}$ is false: the probability of the data given the hypothesis is not the probability of the hypothesis given the data, which is the reversal Bayes' theorem exists to handle and which the probability course named the prosecutor's fallacy. A $p$ -value of 0.03 does not mean a 3 per cent chance the null is true; that number cannot be computed without a prior.

The conventional threshold of 0.05 is Fisher's, offered in 1925 as a convenient line and not as a law of nature. It has no theoretical standing whatever, and treating 0.049 and 0.051 as different in kind is the single most damaging habit in applied statistics.

One tail or two

If the alternative is $p \neq 0.5$ , deviations in either direction count against the null and the $p$ -value sums both tails. If the alternative is one-directional, only one tail counts, and the $p$ -value is half as large for the same data.

The temptation is obvious and the rule is strict: the direction must be chosen before seeing the data, on grounds that would have held whatever the data showed. Choosing one-sided after seeing which way the result went is halving the $p$ -value by fiat, and it converts a nominal 5 per cent error rate into 10 per cent. Two-sided is the default for that reason, and a one-sided test needs a stated justification.

Example. A coin gives 60 heads in 100 tosses. Test $H_{0} : p = 0.5$ against $p \neq 0.5$ , exactly.

Under the null the count is binomial with $n = 100$ and $p = 0.5$ . The upper tail is $P (X \geq 60) = 0.02844$ , and by symmetry the two-sided $p$ -value is $2 (0.02844) = 0.0569$ . At the conventional threshold this does not reach significance, which is worth sitting with: 60 heads out of 100 looks convincing and is not, and the one-sided version would have reported 0.028 and claimed the opposite.

Now you. What would the two-sided $p$ -value be if the same 60 per cent came from 40 heads in a different experiment, tested against $p = 0.5$ with $n = 100$ ? Reason without computing.

Answer

Identical, 0.0569. The binomial with $p = 0.5$ is symmetric about 50, so 40 is exactly as extreme as 60, and a two-sided test cannot distinguish the direction.

An exact test with no approximation

Ronald Fisher's colleague Muriel Bristol claimed she could tell whether milk or tea had been poured into the cup first. Fisher's design, published in 1935, was to prepare eight cups, four each way, tell her that there were four of each, and ask her to identify which four had the milk first.

Under the null hypothesis that she is guessing, every choice of four cups from eight is equally likely, and there are $(\binom{8}{4}) = 70$ of them. Exactly one is entirely correct, so

P (all four correct | guessing) = \frac{1}{70} = 0.0143

Getting three right can happen in $(\binom{4}{3}) (\binom{4}{1}) = 16$ ways, so $P (at least three correct) = 17 / 70 = 0.243$ . Only a perfect score reaches the 5 per cent threshold, and the design says so in advance. Bristol, by the accounts of those present, identified all eight cups correctly.

Three features of this deserve notice. There is no approximation anywhere: the $p$ -value is a count of arrangements, which is why it is called an exact test. The design was fixed before the experiment, including the number of cups and the fact that she was told there were four of each, and the calculation depends on both. And the test has a known and modest resolution: with six cups, three each way, there are $(\binom{6}{3}) = 20$ arrangements, a perfect score gives $p = 0.05$ exactly, and no result whatever could have produced a smaller $p$ -value.

Example. With ten cups, five each way, what is the $p$ -value of a perfect identification?

There are $(\binom{10}{5}) = 252$ ways to choose five cups from ten, so a perfect score has $p = 1 / 252 = 0.0040$ . Adding two cups took the strongest possible evidence from $p = 0.014$ to $p = 0.004$ .

Now you. With eight cups, what is the $p$ -value if she gets exactly three of the four right?

Answer

At least three correct has probability $17 / 70 = 0.243$ , so $p = 0.24$ . Three out of four sounds impressive and is entirely ordinary under guessing.

The $t$ test on real data

Return to the 130 body temperature readings, with $\overline{x} = 98.249$ °F and $s = 0.733$ °F, and test the conventional value $H_{0} : μ = 98.6$ °F.

The estimated standard error is $0.733 / \sqrt{130} = 0.0643$ , so

t = \frac{98.249 - 98.6}{0.0643} = - 5.46

With 129 degrees of freedom the two-sided $p$ -value is $2.4 \times 10^{- 7}$ . Data like this is essentially impossible if the population mean really were 98.6, so the conventional figure is rejected, and the direction and size of the discrepancy are both worth stating: the estimate is 0.35 °F below the convention, or about 0.2 °C.

Notice how much more the interval said than the test. The test reports that 98.6 is implausible; the interval, $(98.12, 98.38)$ , reports that and also that the mean is near 98.25 and pinned down to about a tenth of a degree. Whenever both are available, the interval is the better thing to publish, which is the recommendation of every set of reporting guidelines written in the last thirty years and is still the minority practice.

Example. A sample of 25 has $\overline{x} = 12.4$ and $s = 3.0$ . Test $H_{0} : μ = 11$ against a two-sided alternative.

The standard error is $3.0 / \sqrt{25} = 0.6$ , so $t = (12.4 - 11) / 0.6 = 2.333$ on 24 degrees of freedom, giving a two-sided $p$ -value of 0.028. The null is rejected at the 5 per cent level.

Now you. Test the same sample against $H_{0} : μ = 11.5$ .

Answer

$t = (12.4 - 11.5) / 0.6 = 1.50$ , and the two-sided $p$ -value on 24 degrees of freedom is 0.147. The same data rejects 11 and does not reject 11.5, which is the duality of the next section in action.

Tests and intervals are the same thing

Those two results are not a coincidence. A two-sided test of $H_{0} : μ = μ_{0}$ at level $α$ rejects exactly when $μ_{0}$ lies outside the $1 - α$ confidence interval for $μ$ , and the reason is that both come from the same pivot. The interval is the set of null values that would not be rejected; the test asks whether one particular value is in that set.

For the sample above the 95 per cent interval is $12.4 \pm 2.064 (0.6) = (11.16, 13.64)$ . The value 11 is outside it, so the test rejects; 11.5 is inside, so it does not. Nothing new was computed either time.

The practical consequence is that the interval strictly dominates the test as a report. It answers the test's question, by inspection, for every possible null value at once, and it additionally reports the size and the precision. A paper that reports only $p = 0.03$ has thrown away information it already computed.

What a $p$ -value is not

Five confusions, each responsible for a large literature.

It is not the probability that the null hypothesis is true. It is not the probability that the results are due to chance, which is the same error in casual dress. It is not a measure of the size of the effect: with $n$ large enough, a difference of no practical consequence produces an arbitrarily small $p$ -value, since $t$ grows as $\sqrt{n}$ for any fixed discrepancy. It is not the probability of a replication succeeding. And a non-significant result is not evidence of no effect.

The last two of those are the subject of the next lesson, which asks what happens when the null is false: how often a real effect is detected, how many observations that takes, and why a study that finds significance without enough observations to have deserved it is worse than one that finds nothing at all.

The question a test answers

The test statistic and the p-value

One tail or two

An exact test with no approximation

The t test on real data

Tests and intervals are the same thing

What a p-value is not

The test statistic and the $p$ -value

The $t$ test on real data

What a $p$ -value is not