Rare events and the Poisson law

The binomial formula becomes unusable when the number of trials runs into the thousands and the success probability into the thousandths, which is exactly the regime of accidents, faults, arrivals and decays. This lesson takes the limit, finds a one-parameter law with a one-line formula, and checks it against three sets of real counts, one of which decided a wartime question about whether bombs were being aimed.

Taking the limit

Start from the binomial with $n$ trials and success probability $p$ , and hold the mean fixed at $λ = n p$ while letting $n$ grow, so $p = λ / n$ shrinks. Write out the mass function and group the pieces:

P (X = k) = \frac{n (n - 1) \dots (n - k + 1)}{k!} {(\frac{λ}{n})}^{k} {(1 - \frac{λ}{n})}^{n - k}

= \frac{λ^{k}}{k!} \cdot \frac{n (n - 1) \dots (n - k + 1)}{n^{k}} \cdot {(1 - \frac{λ}{n})}^{n} \cdot {(1 - \frac{λ}{n})}^{- k}

Each of the three trailing factors has an easy limit with $k$ and $λ$ held fixed. The first is a product of $k$ ratios each tending to $1$ , since $(n - j) / n \to 1$ . The second is the definition of the exponential met in the foundations course, $(1 + x / n)^{n} \to e^{x}$ , giving $e^{- λ}$ . The third tends to $1$ because $λ / n$ tends to $0$ and $k$ is fixed. What survives is the Poisson distribution:

P (X = k) = \frac{e^{- λ} λ^{k}}{k!}, k = 0, 1, 2, \dots

named for Siméon Denis Poisson, who published it in 1837 in a book about the reliability of jury verdicts, where it went largely unnoticed for sixty years.

It is a legitimate distribution because $\sum_{k} λ^{k} / k!$ is the series for $e^{λ}$ , so the masses sum to $e^{- λ} e^{λ} = 1$ . There is one parameter, no $n$ and no $p$ separately, only their product. That is the practical payoff: to use it you need the average rate and nothing else, and the average rate is usually the one thing that has been measured.

The convergence is fast. With $n = 1000$ and $p = 0.003$ , so $λ = 3$ , the binomial gives $P (X = 0) = 0.049563$ against the Poisson $0.049787$ , and $P (X = 3) = 0.224379$ against $0.224042$ . Agreement to three decimal places, with the exact calculation requiring $C (1000, 3)$ and the approximation requiring a pocket calculator.

Mean, variance, and a test you can run

Both moments follow from the limit. The binomial mean $n p$ is $λ$ by construction, and the binomial variance $n p (1 - p)$ tends to $λ$ as well, since $p \to 0$ . So

E [X] = λ, operatorname {V a r} (X) = λ

Mean equals variance, which is unusual and useful. It gives a free diagnostic: compute the sample mean and the sample variance of any set of counts, and if they differ substantially the Poisson model is wrong. Counts more spread out than their mean, called overdispersed, indicate clustering, meaning the events are not independent. Counts less spread out indicate some regulating mechanism spacing them apart.

Example. A call centre receives on average $4$ calls a minute. What is the probability of exactly $2$ calls in a given minute, and of none at all?

$P (2) = e^{- 4} 4^{2} / 2! = e^{- 4} \times 8 \approx 0.1465$ . $P (0) = e^{- 4} \approx 0.0183$ , so a completely quiet minute happens about once every fifty-five minutes.

Now you. A shop averages $2.5$ customers per five-minute period. What is the probability that a given five-minute period has none?

Answer

$P (0) = e^{- 2.5} \approx 0.0821$ , about one period in twelve.

Horse kicks

Ladislaus von Bortkiewicz published the first serious test of this law in 1898, in a book whose title translates as The Law of Small Numbers. He took Prussian army records of soldiers killed by horse kicks, for ten cavalry corps over the twenty years from 1875 to 1894: $200$ corps-years, and $122$ deaths in total.

Each corps-year is a huge number of opportunities, since many soldiers stand near many horses on many days, each with a tiny chance of a fatal kick. That is the Poisson regime, with $λ = 122 / 200 = 0.61$ deaths per corps-year. The prediction for the number of corps-years with $k$ deaths is $200 e^{- 0.61} (0.61)^{k} / k!$ :

Deaths in a corps-year	Observed	Poisson prediction
0	109	108.7
1	65	66.3
2	22	20.2
3	3	4.1
4	1	0.6

The fit is close enough to be slightly eerie, and it was obtained from a single number, the average. Nothing about horses, soldiers or Prussia entered the calculation. Whenever a great many independent opportunities each carry a tiny probability, the counts fall into this shape regardless of what the opportunities are.

Bombs on London

In 1946 R. D. Clarke, an actuary, published a two-page paper applying the same test to a question that had mattered a great deal in 1944. During the V-1 flying bomb attacks, south London appeared to be hit in clusters, and the natural inference was that the weapons were being aimed accurately enough to concentrate on particular districts, which had consequences for whether people should be moved.

Clarke divided a $144$ square kilometre area of south London into $576$ squares of a quarter of a square kilometre each, and counted the $537$ hits that fell in it. If the bombs landed at random, the count per square is Poisson with $λ = 537 / 576 = 0.9323$ .

Hits in a square	Observed	Poisson prediction
0	229	226.7
1	211	211.4
2	93	98.5
3	35	30.6
4	7	7.1
5 or more	1	1.6

The agreement is excellent, so the bombs were falling at random and the apparent clusters were what randomness looks like. This is the point worth carrying away. Random scatter is lumpy. A uniform sprinkling with no gaps and no clumps would be evidence of aiming, not of chance, and the human eye reads clumps as design. Under this model $93 + 35 + 7 + 1 = 136$ of the $576$ squares should take two or more hits, and $136$ did.

Example. Using Clarke's $λ = 0.9323$ , what fraction of squares would be expected to take no hits at all, and how many of the $576$ is that?

$P (0) = e^{- 0.9323} \approx 0.3937$ , so $576 \times 0.3937 \approx 226.7$ squares, against $229$ observed. Nearly two in five squares escape entirely, which is what makes the hit squares look concentrated.

Now you. A different area is divided into $400$ squares taking $600$ hits in total. How many squares would be expected to take no hits?

Answer

$λ = 600 / 400 = 1.5$ , so $P (0) = e^{- 1.5} \approx 0.2231$ and the expected count is $400 \times 0.2231 \approx 89.3$ squares.

Counting in time

The two examples above counted events in space and in years, and the general structure has a name. A Poisson process of rate $λ$ per unit of time is a stream of events with three properties: the number in any interval depends only on its length, counts in disjoint intervals are independent, and events do not coincide. Chop an interval of length $t$ into $n$ tiny slices, each holding an event with probability about $λ t / n$ , and the limit above gives a Poisson count with parameter $λ t$ .

Rutherford, Geiger and Bateman tested this in 1910 on radioactive decay, counting alpha particles from a polonium source in $2608$ intervals of $7.5$ seconds each. They recorded $10097$ particles, a mean of $3.87$ per interval, against which the Poisson prediction runs $54.3$ , $210.3$ , $407.1$ , $525.3$ , $508.4$ for zero through four particles, against observed counts of $57$ , $203$ , $383$ , $525$ , $532$ . Radioactive decay is the cleanest Poisson process known, because nuclei genuinely do not influence one another and genuinely do not age.

Example. A web server receives requests as a Poisson process at $3$ per second. What is the probability of at least $3$ requests in a given second?

$P (0) + P (1) + P (2) = e^{- 3} (1 + 3 + 4.5) = e^{- 3} \times 8.5 \approx 0.4232$ , so $P (\geq 3) \approx 0.5768$ .

Now you. For the same server, what is the probability of at least $2$ requests in a half second interval? Note that halving the interval halves the parameter.

Answer

Here $λ = 1.5$ , so $P (0) + P (1) = e^{- 1.5} (1 + 1.5) \approx 0.5578$ and $P (\geq 2) \approx 0.4422$ .

Where the model breaks

The Poisson law assumes independence and a constant rate, and real count data violates both routinely.

Traffic accidents at a junction cluster, because one crash causes the next. Insurance claims from storms cluster, because one storm causes many. Goals in football arrive at a rate that changes with the score. In each case the counts are overdispersed, with variance well above the mean, and using a Poisson model understates the chance of a very bad period, which is precisely the number an insurer or a hospital needs. The mean-equals-variance check catches this in one calculation, which is why it is worth doing before anything else.

There is also a limit of a different kind, and it applies to everything in the course so far. Every distribution met up to here is discrete: the variable takes values in a list, and probability is assigned to individual values. Ask for the exact time until the next alpha particle rather than the count in an interval, or the exact position of a bomb rather than which square it fell in, and the list becomes a continuum. Assigning a positive probability to each of uncountably many values is impossible, since they would sum to more than one, and yet one of those values does occur.

Resolving that, by replacing the sum with an area and the mass function with a density, is the next lesson.