The binomial distribution

A distribution earns a name when one mechanism produces it over and over, and the commonest mechanism of all is a fixed number of independent attempts at the same thing. This lesson derives what that produces, tests it against a dice experiment run in 1894, and finds the regime where the formula stops being usable.

One trial, two outcomes

Start with the smallest case. A Bernoulli trial is an experiment with two outcomes, labelled success and failure, with $P (success) = p$ . Its indicator $X$ takes the value $1$ or $0$ , and everything about it follows from the previous two lessons:

E [X] = 1 \cdot p + 0 \cdot (1 - p) = p, E [X^{2}] = 1 \cdot p + 0 \cdot (1 - p) = p

so $operatorname {V a r} (X) = p - p^{2} = p (1 - p)$ . That variance is worth a moment. It is zero at $p = 0$ and at $p = 1$ , where the outcome is certain and there is nothing to vary, and it is largest at $p = 0.5$ , where it equals $0.25$ . Maximum uncertainty sits at even odds, which matches intuition and is here a consequence rather than an assertion. Writing $q = 1 - p$ for the failure probability, the variance is $p q$ .

The name is Jacob Bernoulli's, from Ars Conjectandi, written before his death in 1705 and published in 1713, which is also where the first law of large numbers appears.

Counting successes

Now run $n$ such trials, independently, with the same $p$ each time, and let $X$ be the number of successes. Any particular sequence with $k$ successes and $n - k$ failures has probability $p^{k} q^{n - k}$ by independence, and this does not depend on the order in which the successes fell. So the probability of $k$ successes is that quantity times the number of sequences that have $k$ successes, which is the number of ways to choose which trials succeeded:

P (X = k) = C (n, k) p^{k} q^{n - k}, k = 0, 1, \dots, n

This is the binomial distribution, written $X \sim Bin (n, p)$ , and both ingredients came from earlier lessons: the counting of positions from the combinations lesson, the multiplication of probabilities from independence. The masses sum to $1$ because the right-hand side is the general term of the binomial theorem applied to $(p + q)^{n} = 1^{n} = 1$ , which is where the name comes from.

Toss a fair coin ten times. The probability of exactly three heads is $C (10, 3) (0.5)^{10} = 120 / 1024 \approx 0.1172$ , and the probability of eight or more is $(45 + 10 + 1) / 1024 = 56 / 1024 \approx 0.0547$ . That last figure is worth remembering: getting eight or more heads out of ten is not rare enough to prove anything about a coin.

Three conditions are being assumed, and each fails somewhere real. The number of trials is fixed in advance, so stopping when you like invalidates the formula. The trials are independent, which fails for cards dealt without replacement. And $p$ is the same throughout, which fails when a machine drifts as it heats up.

Example. A machine produces items with a $5$ percent defect rate, independently. In a box of $20$ , what is the probability of no defects, and of two or more?

$P (0) = {0.95}^{20} \approx 0.3585$ . $P (1) = 20 (0.05) (0.95)^{19} \approx 0.3774$ . So $P (\geq 2) = 1 - 0.3585 - 0.3774 \approx 0.2642$ . More than a quarter of boxes carry at least two defects, even though the expected number is only $1$ .

Now you. For the same machine, what is the probability that a box of $10$ contains no defects?

Answer

${0.95}^{10} \approx 0.5987$ , so just under $60$ percent of small boxes are clean.

Mean and variance without a single sum

The direct route to $E [X]$ is $\sum_{k} k C (n, k) p^{k} q^{n - k}$ , which can be done and is unpleasant. The indicator route is two lines.

Write $X = I_{1} + I_{2} + \dots + I_{n}$ , where $I_{j}$ indicates success on trial $j$ . Linearity of expectation, which needs nothing at all, gives

E [X] = n p

Independence lets the variances add too, and each contributes $p q$ :

operatorname {V a r} (X) = n p q, σ = \sqrt{n p q}

Ten fair tosses have mean $5$ and standard deviation $\sqrt{10 \times 0.25} \approx 1.58$ . The box of twenty items has mean $1$ and standard deviation $\sqrt{20 (0.05) (0.95)} \approx 0.9747$ .

The shape follows from the ratio of consecutive masses,

\frac{P (X = k)}{P (X = k - 1)} = \frac{(n - k + 1) p}{k q}

which decreases as $k$ grows, so the masses rise while that ratio exceeds $1$ and fall afterwards. The distribution therefore has a single peak, located at $k = ⌊ (n + 1) p ⌋$ , close to the mean $n p$ as it should be. At $p = 0.5$ the mass function is symmetric, since swapping successes for failures leaves it unchanged, and away from $0.5$ it is skewed towards the boundary it is furthest from.

Weldon's twenty-six thousand throws

In 1894 Raphael Weldon threw twelve dice together, over and over, and recorded how many showed a five or a six. He did this $26306$ times, which is $315672$ individual dice rolls, and sent the tally to Karl Pearson. On fair dice the probability of a five or six is $1 / 3$ , so each throw of twelve should give $Bin (12, 1 / 3)$ , with mean $4$ successes.

The observed total was $106602$ successes, an average of $4.0524$ per throw, giving $\hat{p} = 106602 / 315672 \approx 0.33771$ against the theoretical $0.33333$ . The excess looks tiny. Ask whether it could be chance, using the machinery just built: if $p$ were exactly $1 / 3$ , the total number of successes would have mean $315672 / 3 = 105224$ and standard deviation $\sqrt{315672 \times \frac{13}{\times} \frac{23}{}} \approx 264.9}$ . The observed total is $1378$ above the mean, which is $5.2$ standard deviations out. Chebyshev alone caps that at $1 / {5.2}^{2} \approx 0.037$ , and the true figure is vastly smaller.

So the dice were not fair, and the reason is manufacturing: the pips are drilled into the faces, removing material, so the six face is lightest and lands uppermost slightly more often than a third of the time. This is what a probability model earns its keep by doing. A model precise enough to be wrong turned a $0.4$ percentage point discrepancy, invisible to any player, into a definite statement about how dice are made.

Example. A drug is claimed to be no better than a coin toss at curing a condition. In a trial of $8$ patients it cures $7$ . Under the coin-toss hypothesis, how likely is a result this extreme or more?

$P (X \geq 7) = (C (8, 7) + C (8, 8)) / 2^{8} = (8 + 1) / 256 \approx 0.0352$ . About one trial in twenty-eight would do this well by luck alone, which is suggestive and thin, and it is why trials are not run on eight patients.

Now you. In a trial of $10$ patients the drug cures $9$ . Under the same hypothesis, how likely is nine or more?

Answer

$(C (10, 9) + C (10, 10)) / 2^{10} = 11 / 1024 \approx 0.0107$ , about one in ninety-three.

Waiting instead of counting

The binomial fixes the number of trials and lets the number of successes vary. Turn it around: fix the goal at one success and let the number of trials vary. Let $Y$ be the trial on which the first success occurs. The first $k - 1$ trials must fail and the $k$ th must succeed, so

P (Y = k) = q^{k - 1} p, k = 1, 2, 3, \dots

the geometric distribution. Its masses form a geometric series with first term $p$ and ratio $q$ , summing to $p / (1 - q) = 1$ , so it is a legitimate distribution on an infinite sample space, the one promised in the second lesson.

Its mean can be found by a trick that avoids summing $\sum k q^{k - 1} p$ . Condition on the first trial. With probability $p$ it succeeds and $Y = 1$ . With probability $q$ it fails, and the situation afterwards is identical to the start, so the remaining wait has the same expectation $E [Y]$ . Hence

E [Y] = p (1) + q (1 + E [Y]) = 1 + q E [Y] ⟹ E [Y] = \frac{1}{p}

Waiting for a six on a fair die takes $6$ rolls on average; waiting for a lottery win at one in $13983816$ takes that many draws. The same self-similarity argument gives the variance, $q / p^{2}$ .

That self-similarity is the geometric distribution's defining property, and it is called memorylessness: given that the first $m$ trials failed, the remaining wait has exactly the original distribution. A machine that has run for a thousand hours without failing is, under this model, as good as new. That is sometimes true, for failures caused by external shocks, and often false, for anything that wears out. Noticing which case applies is a modelling decision the mathematics cannot make.

Example. A basketball player scores each free throw with probability $0.8$ , independently. What is the probability that their first miss is on the fourth attempt, and how many attempts until the first miss on average?

A miss is the "success" here, with $p = 0.2$ . So $P (Y = 4) = {0.8}^{3} \times 0.2 = 0.1024$ , and $E [Y] = 1 / 0.2 = 5$ attempts.

Now you. A fair die is rolled until a six appears. What is the probability that this takes exactly three rolls?

Answer

$(5 / 6)^{2} (1 / 6) = 25 / 216 \approx 0.1157$ .

Where the formula gives out

The binomial is exact, and it becomes unusable in a regime that occurs constantly.

Consider a factory making $10000$ components a day, each defective with probability $0.0003$ , or a city of a hundred thousand people each having a rare accident on a given day, or a stretch of DNA with a per-base mutation chance in the millionths. These are binomials with $n$ enormous and $p$ minuscule. The formula demands $C (10000, k)$ , which for moderate $k$ is a number with hundreds of digits, multiplied by $p^{k}$ , which is vanishingly small, and the two nearly cancel. Computed directly in ordinary arithmetic this overflows and then underflows, and even done carefully it obscures what is going on.

Look at what stays fixed as the regime is entered. The mean $n p$ is a sensible number, three defective components a day, and the variance $n p q$ is nearly the same, since $q \approx 1$ . Both depend on $n$ and $p$ only through their product, which suggests the distribution itself might too, in the limit where $n$ grows and $p$ shrinks with $n p$ held constant.

It does, and the limiting law has a one-line formula, a single parameter, and a striking record against real count data. It is the next lesson.