Comparing two groups

Almost no real question is whether a mean equals a fixed number. The questions are whether treated patients do better than untreated ones, whether a change to a process improved it, whether two populations differ. All of those compare two groups, and the arithmetic depends on how the two groups were formed, which is a fact about the design rather than about the data.

Two designs, and why the difference matters

In a paired design each observation in one group has a natural partner in the other: the same patient before and after, two plants in the same pot, matched twins. In an independent samples design the two groups are separate sets of units with no correspondence between them.

The distinction is not a formality, because pairing removes a source of variation. If plants differ enormously from pot to pot but the two plants within a pot are alike, then comparing group means drowns the treatment effect in pot-to-pot variation, while comparing within pots does not. Analysing paired data as though it were independent throws that advantage away and usually turns a clear result into a null one. Analysing independent data as though it were paired is not possible, since there are no pairs, but pairing on something irrelevant wastes degrees of freedom for nothing.

The rule is simple: pair when the units genuinely come in linked couples, and let the design decide the analysis rather than the other way round.

Paired data reduces to one sample

The trick with paired data is that it is not a two-sample problem at all. Form the difference $d_{i} = x_{i} - y_{i}$ within each pair and the question becomes whether those differences have mean zero, which is the one-sample $t$ test of the previous lessons applied to $n$ differences.

Charles Darwin measured this in 1876. He raised 15 pairs of Zea mays, one cross-fertilised and one self-fertilised plant in each pot, and recorded their final heights in eighths of an inch. The differences, cross minus self, were

49, -67, 8, 16, 6, 23, 28, 41, 14, 29, 56, 24, 75, 60, -48

Their mean is 20.93 eighths, about 2.62 inches, with $s = 37.74$ . The standard error is $37.74 / \sqrt{15} = 9.75$ , so

t = \frac{20.93}{9.75} = 2.148

on 14 degrees of freedom, giving a two-sided $p$ -value of 0.0497. Fisher used exactly this dataset to introduce the paired $t$ test, and it is an instructive choice: the result scrapes past 0.05 by three thousandths, two of the fifteen differences are strongly negative, and the 95 per cent interval for the mean difference runs from 0.03 to 41.8 eighths of an inch. The evidence for an effect is real and the size of it is barely constrained at all. Reporting only " $p < 0.05$ " would conceal that completely.

Example. Ten runners are timed before and after a training programme, and their improvements in seconds have mean 3.2 with standard deviation 4.0. Test whether the programme had any effect.

The standard error is $4.0 / \sqrt{10} = 1.265$ , so $t = 3.2 / 1.265 = 2.53$ on 9 degrees of freedom, and the two-sided $p$ -value is 0.032. There is evidence of improvement, though with ten runners and no control group the improvement could be practice, fitness drift or regression to the mean rather than the programme.

Now you. What is the 95 per cent confidence interval for the mean improvement, given $t_{0.975, 9} = 2.262$ ?

Answer

$3.2 \pm 2.262 (1.265) = 3.2 \pm 2.86$ , so $(0.34, 6.06)$ seconds. The interval excludes zero, which is the same conclusion as the test, and it also shows the effect could be anything from trivial to substantial.

Two independent samples

With no pairing, the statistic is the difference of the two sample means, ${\overline{X}}_{1} - {\overline{X}}_{2}$ . Its expectation is $μ_{1} - μ_{2}$ by linearity, and since the samples are independent, variances add:

Var ({\overline{X}}_{1} - {\overline{X}}_{2}) = \frac{σ_{1}^{2}}{n_{1}} + \frac{σ_{2}^{2}}{n_{2}}

Variances add even though the means subtract, which is the point most often got wrong. Subtracting two noisy quantities produces something noisier than either.

If the two populations can be assumed to share a variance $σ^{2}$ , both samples estimate it, and the efficient move is to combine them into the pooled variance

s_{p}^{2} = \frac{(n_{1} - 1) s_{1}^{2} + (n_{2} - 1) s_{2}^{2}}{n_{1} + n_{2} - 2}

a weighted average with weights equal to the degrees of freedom each sample contributes. Then

t = \frac{{\overline{x}}_{1} - {\overline{x}}_{2}}{s_{p} \sqrt{\frac{1}{n_{1}} + \frac{1}{n_{2}}}}

on $n_{1} + n_{2} - 2$ degrees of freedom. The divisor is $n_{1} + n_{2} - 2$ because two means were estimated before the residuals were formed, exactly as the $n - 1$ lesson predicted.

Example. Group A has 12 observations with mean 24.5 and $s = 3.2$ ; group B has 15 with mean 21.0 and $s = 2.8$ . Test for a difference and give the interval.

The pooled variance is $\frac{11 (10.24) + 14 (7.84)}{25} = 8.896$ , so $s_{p} = 2.983$ . The standard error of the difference is $2.983 \sqrt{1 / 12 + 1 / 15} = 1.155$ , giving $t = 3.5 / 1.155 = 3.03$ on 25 degrees of freedom and $p = 0.0056$ . The 95 per cent interval for the difference is $3.5 \pm 2.060 (1.155) = (1.12, 5.88)$ .

Now you. What is the standardised effect size, Cohen's $d = ({\overline{x}}_{1} - {\overline{x}}_{2}) / s_{p}$ ?

Answer

$3.5 / 2.983 = 1.17$ , an effect of well over one standard deviation, which by the conventional labels is large. The groups are separated by more than they are spread.

When the variances are not equal

Pooling assumes a common variance, and when that fails with unequal sample sizes the test misbehaves badly. Welch's version drops the assumption, estimating each variance separately:

t = \frac{{\overline{x}}_{1} - {\overline{x}}_{2}}{\sqrt{\frac{s_{1}^{2}}{n_{1}} + \frac{s_{2}^{2}}{n_{2}}}}

with a fractional degrees of freedom given by the Welch-Satterthwaite formula, which is what software reports when it prints something like "df = 47.24".

The difference is not cosmetic. Take group A with 10 observations, mean 50 and $s = 2$ , and group B with 40 observations, mean 46 and $s = 10$ . The pooled calculation is dominated by B's large variance, gives a standard error of 3.20 and $t = 1.25$ , and reports $p = 0.22$ . Welch gives a standard error of 1.70, $t = 2.35$ on 47.2 degrees of freedom, and $p = 0.023$ . The two analyses of the same data disagree about the conclusion, and Welch is the correct one, because pooling has assigned B's variability to A's small sample where it does not belong.

Welch's test costs almost nothing when the variances really are equal, so the sensible default is to use it always rather than to test for equal variances first and choose. Testing for equality of variance and then picking the test based on the outcome is a two-stage procedure whose overall error rate is not the nominal one, which is the same objection that applies to any analysis chosen after seeing the data.

Two proportions, and a trial that settled something

For binary outcomes the comparison is between ${\hat{p}}_{1}$ and ${\hat{p}}_{2}$ , and the standard error of the difference again comes from adding variances:

SE = \sqrt{\frac{{\hat{p}}_{1} (1 - {\hat{p}}_{1})}{n_{1}} + \frac{{\hat{p}}_{2} (1 - {\hat{p}}_{2})}{n_{2}}}

The 1954 field trial of the Salk polio vaccine is the canonical example, and it was run as a randomised double-blind placebo-controlled experiment precisely because earlier uncontrolled evidence had been unreadable. In the placebo-control portion, 200,745 children received the vaccine and 33 developed paralytic polio; 201,229 received a placebo and 115 did.

The rates are $33 / 200 {,} 745 = 16.4$ per 100,000 and $115 / 201 {,} 229 = 57.1$ per 100,000, a difference of 40.7 per 100,000. The standard error of that difference is 6.05 per 100,000, so the 95 per cent interval runs from 28.9 to 52.6 per 100,000 and $z = 6.7$ , which corresponds to a $p$ -value far below any threshold anyone uses.

The same result stated three ways is worth having in view at once. The risk difference is 40.7 per 100,000, from which the number needed to treat is $1 / 0.000407 = 2456$ children vaccinated to prevent one paralytic case. The relative risk is $16.4 / 57.1 = 0.288$ , so vaccination cut the rate to under a third, an efficacy of 71 per cent. Both describe the same two counts. The relative figure sounds far more impressive than the absolute one, and which of them is the honest headline depends entirely on how common the disease is, a point the final lesson returns to.

Example. In a trial, 40 of 500 in the treatment group and 60 of 500 in the control group had the outcome. Compute the risk difference and its standard error.

${\hat{p}}_{1} = 0.08$ and ${\hat{p}}_{2} = 0.12$ , so the difference is 0.04. The standard error is $\sqrt{0.08 (0.92) / 500 + 0.12 (0.88) / 500} = 0.0189$ , giving $z = 2.11$ and a two-sided $p$ -value of 0.035.

Now you. What is the relative risk, and how many patients must be treated to prevent one event?

Answer

The relative risk is $0.08 / 0.12 = 0.667$ , a one-third reduction. The number needed to treat is $1 / 0.04 = 25$ .

Sample size for a two-group study

Because the difference of two means has variance $σ^{2} (1 / n_{1} + 1 / n_{2})$ , which for equal groups of size $n$ is $2 σ^{2} / n$ , the sample size formula from the previous lesson picks up a factor of two:

n per group = \frac{2 (z_{1 - α / 2} + z_{1 - β})^{2}}{d^{2}} = \frac{15.70}{d^{2}}

at $α = 0.05$ and 80 per cent power. So detecting a half-standard-deviation difference needs 63 per group, and a fifth of a standard deviation needs 393 per group. A two-group comparison costs about four times the total sample of a one-sample test for the same effect: twice for having two groups and twice again because the noise of two estimates adds.

Equal allocation is optimal when the variances are equal, and when they differ the optimal split is proportional to the standard deviations, which is the Neyman allocation from the sampling lesson reappearing in a different costume.

What the tests assume

The $t$ tests assume independent observations within and between groups, and approximate normality of the sampling distribution of each mean, which the central limit theorem supplies unless the samples are small and skewed. They do not assume the data is normal when $n$ is comfortably large, and the fourth lesson's exact tail calculations say how large "comfortably" is: sooner for symmetric data, much later for skewed.

Two failures matter more than non-normality. Dependence between observations, from clustering, repeated measures on the same subject, or time-series structure, inflates the true variance while the formula stays the same, so the standard error is too small and the $p$ -value is too. And a difference between groups is a difference between whatever else distinguishes them: if the treated group was sicker to begin with, the $t$ test does not know and will attribute the whole gap to the treatment.

That second failure is a different kind of problem from anything in this lesson, and no test repairs it. It is why the Salk trial randomised, and it is the subject of two lessons from now. Before that, the remaining outcome type: data that is not a measurement at all, but a count in a category.