Counts and categories

Everything so far assumed the outcome was a number that could be averaged. A great deal of data is not: blood group, party voted for, seed shape, whether a component failed. The natural summary is a table of counts, and the natural question is whether those counts match what some model predicts. This lesson builds the standard test for that, on data from Gregor Mendel's garden, and then turns the same test on Mendel himself.

Comparing counts with a model

Suppose $n$ items fall into $k$ categories, with observed counts $O_{1}, dots, O_{k}$ , and a model predicts probabilities $π_{1}, dots, π_{k}$ , so the expected counts are $E_{i} = n π_{i}$ . Some discrepancy is certain, and the question is whether this much discrepancy is ordinary.

Each count is binomial with mean $E_{i}$ and variance $n π_{i} (1 - π_{i})$ , so a natural standardised deviation is $(O_{i} - E_{i}) / \sqrt{E_{i}}$ , using $E_{i}$ rather than the full binomial variance in the denominator. Squaring and summing gives Karl Pearson's 1900 statistic

X^{2} = \sum_{i = 1}^{k} \frac{(O_{i} - E_{i})^{2}}{E_{i}}

Two features are worth noticing before any distribution theory. Dividing by $E_{i}$ rather than by $E_{i}^{2}$ makes the statistic scale with $n$ : doubling all the counts while keeping the proportions fixed doubles $X^{2}$ , which is what a measure of evidence should do. And squaring means direction is lost, so the test cannot say which way the data departed, only that it did.

Pearson showed that under the model, for large enough counts, $X^{2}$ follows a chi-squared distribution. That distribution is the sum of squares of independent standard normal variables, and its single parameter, the degrees of freedom, is the number of independent squares being added.

Degrees of freedom by counting constraints

The degrees of freedom are not the number of categories, and the reason is a counting argument that works in every case in this lesson.

Start with $k$ deviations $O_{i} - E_{i}$ . They are not free to vary independently, because the counts must total $n$ , which forces $\sum_{i} (O_{i} - E_{i}) = 0$ : one constraint. If the model itself had parameters estimated from the same data, each estimated parameter imposes a further constraint. So

df = k - 1 - (number of parameters estimated from the data)

For a fully specified model such as Mendel's 3 to 1 ratio, nothing is estimated and the degrees of freedom are $k - 1$ . For a Poisson model whose rate was estimated from the same counts, it is $k - 2$ .

The same argument gives the degrees of freedom for a two-way table, later in this lesson, and it is the same argument as the $n - 1$ in the sample variance: each quantity estimated from the data removes one dimension from the space the residuals can occupy.

Goodness of fit, in Mendel's garden

Mendel's 1866 paper reports that crossing pea plants heterozygous for seed shape gave 5474 round seeds and 1850 wrinkled, a total of 7324. His theory predicts a 3 to 1 ratio, so the expected counts are $0.75 (7324) = 5493$ and $0.25 (7324) = 1831$ .

X^{2} = \frac{(5474 - 5493)^{2}}{5493} + \frac{(1850 - 1831)^{2}}{1831} = \frac{361}{5493} + \frac{361}{1831} = 0.0657 + 0.1972 = 0.263

With $k = 2$ categories and no parameters estimated, there is 1 degree of freedom, and the 5 per cent critical value is 3.841. The observed 0.263 is far below it, with $p = 0.61$ : the data is entirely consistent with 3 to 1. That is the correct and modest conclusion. The test has not proved the ratio is exactly 3 to 1, since a 2.9 to 1 ratio would also fit comfortably at this sample size; it has failed to find any discrepancy worth reporting.

Example. Mendel's dihybrid cross of round-yellow with wrinkled-green gave 315, 108, 101 and 32 seeds in the four phenotype classes, a total of 556, against a predicted 9:3:3:1. Test the fit.

The expected counts are $556 \times 9 / 16 = 312.75$ , then 104.25, 104.25 and 34.75. The contributions are $(2.25)^{2} / 312.75 = 0.0162$ , $(3.75)^{2} / 104.25 = 0.1349$ , $(3.25)^{2} / 104.25 = 0.1013$ and $(2.75)^{2} / 34.75 = 0.2176$ , summing to $X^{2} = 0.470$ on 3 degrees of freedom, where the 5 per cent critical value is 7.815. The $p$ -value is 0.93.

Now you. A die is rolled 600 times and the six faces come up 95, 102, 108, 96, 110 and 89 times. Test whether it is fair.

Answer

Each expected count is 100, so $X^{2} = (25 + 4 + 64 + 16 + 100 + 121) / 100 = 3.30$ on 5 degrees of freedom. The critical value is 11.07 and the $p$ -value is 0.65, so there is no evidence of unfairness.

Independence in a two-way table

The second use of the same statistic asks whether two categorical variables are related. Cross-classify $n$ items by two variables into an $r$ by $c$ table with row totals $R_{i}$ and column totals $C_{j}$ . Under the null hypothesis that the two classifications are independent, the probability of landing in cell $(i, j)$ is the product of the marginal probabilities, so the expected count is

E_{i j} = \frac{R_{i} C_{j}}{n}

which is the multiplication rule for independent events, estimated from the margins. Then $X^{2}$ is computed cell by cell as before.

The degrees of freedom follow from the same counting. There are $r c$ cells, one constraint from the total, and the marginal probabilities were estimated from the data: $r - 1$ free row probabilities and $c - 1$ free column ones. So $df = r c - 1 - (r - 1) - (c - 1) = (r - 1) (c - 1)$ . For a 2 by 2 table that is 1, and for a 3 by 2 table it is 2.

For a 2 by 2 table there is a pleasant consistency check: the chi-squared test of independence and the two-proportion $z$ test of the previous lesson are the same test, with $X^{2} = z^{2}$ exactly when the pooled standard error is used. The Salk trial gives $z = 6.727$ and $X^{2} = 45.25$ , and ${6.727}^{2} = 45.25$ .

Example. Of 500 treated patients 40 had the outcome and of 500 controls 60 did. Test independence of treatment and outcome.

The margins are 500 and 500 by rows, 100 and 900 by columns, with $n = 1000$ , so every expected count in the outcome column is $500 (100) / 1000 = 50$ and in the other column 450. Then

X^{2} = \frac{100}{50} + \frac{100}{450} + \frac{100}{50} + \frac{100}{450} = 4.444

on 1 degree of freedom, giving $p = 0.035$ . Squaring the pooled $z$ of 2.108 from the previous lesson gives the same 4.444.

Now you. Three clinics report 30, 25 and 15 successes out of 50 patients each. Compute the expected count of successes per clinic and the degrees of freedom.

Answer

The total is 70 successes out of 150, so each clinic's expected successes are $50 (70) / 150 = 23.33$ and expected failures $50 (80) / 150 = 26.67$ . The table is 3 by 2, so there are $(3 - 1) (2 - 1) = 2$ degrees of freedom. The statistic works out to 9.375, giving $p = 0.009$ .

Where the approximation fails

Chi-squared is an approximation, based on treating each standardised deviation as normal, and it fails when the counts are small. The common rule is that every expected count should be at least 5, with a few allowed as low as 1 provided most are above 5. Note that the rule concerns the expected counts, not the observed ones: a cell with zero observations is fine if its expectation is 20 and a serious problem if its expectation is 0.4, because a tiny denominator makes $X^{2}$ explode.

When the rule is violated there are two repairs. Categories can be merged, provided the merging is decided on subject-matter grounds and not by looking at which merge gives the desired answer. Or the exact test can be used: for a 2 by 2 table, Fisher's exact test conditions on both sets of margins and counts arrangements, exactly as the lady tasting tea did in the testing lesson, and produces a $p$ -value with no approximation at all. Modern software runs it on tables far larger than Fisher could, and there is little reason to use the approximation for a small table.

Yates's continuity correction, which subtracts 0.5 from each $| O - E |$ before squaring, is an older repair for 2 by 2 tables. It makes the test conservative, sometimes markedly so, and where it matters the exact test is better.

Turning the test on Mendel

Mendel's individual results fit beautifully, and so does every other experiment in his paper, and that is itself a fact to be tested. Fisher did so in 1936, combining the chi-squared statistics across all of Mendel's reported experiments. Chi-squared statistics from independent tests add, and so do their degrees of freedom, so the combined value can be assessed in the same way.

The combined statistic was 41.6 on 84 degrees of freedom. Its expected value under the model is the degrees of freedom itself, 84, so the observed fit is less than half as discrepant as it should be, and the probability of a fit at least this close is about 0.00003. Data agreeing with theory this well happens about three times in a hundred thousand.

What that means has been argued for ninety years. Fisher suggested an assistant who knew the expected ratios; others have proposed unconscious stopping rules, in which counting continued until the ratio looked right, or the reporting of representative rather than complete results. What is not in dispute is the statistical point, and it is the one worth carrying away: a test can be failed from either end. A fit that is too good is evidence against the data-generating story just as a fit that is too bad is, and only a test that looks at both tails will notice.

Example. Ten independent goodness-of-fit tests each on 3 degrees of freedom give chi-squared values summing to 12.0. What does that suggest?

The combined statistic is 12.0 on 30 degrees of freedom, whose expectation is 30. A value that low has $P (X^{2} \leq 12.0) = 0.0014$ , so the collection of fits is suspiciously good, and the natural questions are about selective reporting or a shared error rather than about the model being wrong.

Now you. What is the expected value of a chi-squared statistic on 84 degrees of freedom, and what does an observed 41.6 say?

Answer

The expectation is 84. Observing 41.6 means the summed squared deviations are about half what independent random sampling should produce, which is what makes Mendel's data look too regular rather than wrong.

What the test does not say

Chi-squared reports that observed counts and a model disagree by more than chance comfortably allows. It does not say how, since squaring discarded the direction, so the residuals $(O_{i} - E_{i}) / \sqrt{E_{i}}$ should always be inspected to see which cells drove the result. It does not say how much, since the statistic grows with $n$ : a trivial departure from independence becomes significant in a large enough table, so a measure of association such as Cramér's $V$ belongs beside the $p$ -value.

And it says nothing whatever about causation. A significant table shows that two classifications are associated in this sample, which is compatible with either one causing the other, with both being caused by a third variable, or with the sample having been assembled in a way that manufactures the association. Two lessons from now those alternatives are treated properly. First, the case where the predictor is not a category at all but a number.