Continuous random variables

Every distribution so far has assigned probability to individual values in a list. A measurement does not come in a list: a waiting time, a length, a temperature can take any value in an interval, and there are uncountably many of them. This lesson works out what probability has to look like on a continuum, and derives two laws that are already implicit in earlier lessons.

Why no single value can have probability

Suppose a needle is spun and lands at an angle anywhere in $[0, 360)$ degrees, with no direction favoured. What is the probability that it lands at exactly $90$ degrees?

Whatever answer $c$ you give must be the same for every angle, by symmetry. If $c > 0$ , then any $n$ distinct angles form a disjoint collection with total probability $n c$ , and choosing $n$ larger than $1 / c$ makes that exceed one, contradicting the axioms. So $c = 0$ . Every individual angle has probability zero, and yet the needle lands somewhere, so an outcome with probability zero has just occurred.

This is not a paradox, only a consequence of countable additivity not extending to uncountable collections. The complement rule survives intact: an event of probability zero is not impossible, and an event of probability one is not certain. The second lesson flagged this and here is where it bites.

If single values carry nothing, then probability has to be carried by intervals. For the needle, symmetry says the probability of landing in an interval is proportional to its length, and $[0, 360)$ has probability one, so an interval of width $w$ degrees has probability $w / 360$ . Probability has become a measure of size, which is what makes the continuous theory work.

Density and area

Write the interval probability as a rate times a width. For the needle the rate is $1 / 360$ per degree, constant across the range. In general the rate varies, and the function giving it is the probability density function $f (x)$ , defined so that

P (a \leq X \leq b) = the area under f between a and b

which in the notation of calculus is $\int_{a}^{b} f (x) d x$ . The density is not a probability, and it may exceed one: a variable confined to $[0, 0.1]$ and spread evenly has density $10$ per unit, and the total area is still $0.1 \times 10 = 1$ . What a density has are units, namely probability per unit of $x$ . Reading $f (x)$ as "the probability of $x$ " is the standard beginner's error, and the units are what expose it.

Two requirements make a function a density: it is never negative, and the total area under it is exactly $1$ . Those are the continuous forms of the two conditions on a mass function.

The simplest case is the uniform distribution on $[a, b]$ , with $f (x) = 1 / (b - a)$ across that interval and zero outside. Its areas are rectangles, so every probability is a ratio of lengths: for a variable uniform on $[2, 8]$ , $P (3 \leq X \leq 5) = 2 / 6 \approx 0.3333$ , and $P (X > 6) = 2 / 6$ as well. Since single points contribute no area, it makes no difference whether the endpoints are included, and $P (X < 5)$ and $P (X \leq 5)$ are the same number. That convenience is unique to the continuous case.

The function that works for both

Densities do not exist for discrete variables and masses do not exist for continuous ones, but one object exists for every random variable whatsoever. The cumulative distribution function is

F (x) = P (X \leq x)

It runs from $0$ far to the left to $1$ far to the right, never decreases, and gives interval probabilities by subtraction: $P (a < X \leq b) = F (b) - F (a)$ . For a discrete variable it is a staircase, jumping at each value by that value's mass. For a continuous one it is a continuous climb, and the density is its rate of change, so density and cumulative function carry the same information in two forms.

The cumulative function is the object to reach for when a distribution has to be handled numerically, since published tables and software routines nearly always give $F$ rather than $f$ . It is also the natural home of quantiles: the value $x_{q}$ with $F (x_{q}) = q$ splits the distribution so that a fraction $q$ lies below. The median is the case $q = 0.5$ .

Example. A bus arrives at a time uniform on the interval from $0$ to $10$ minutes after you reach the stop. Write down the cumulative function and use it to find the probability of waiting between $3$ and $7$ minutes.

The density is $1 / 10$ per minute, so $F (x) = x / 10$ for $x$ in $[0, 10]$ , zero below and one above. Then $P (3 < X \leq 7) = F (7) - F (3) = 0.7 - 0.3 = 0.4$ .

Now you. For the same bus, find the probability of waiting more than $8$ minutes, and find the median wait.

Answer

$P (X > 8) = 1 - F (8) = 1 - 0.8 = 0.2$ . The median solves $x / 10 = 0.5$ , giving $5$ minutes.

Mean and spread by area

Expectation was a sum of value times mass. On a continuum it becomes a sum of value times density times width, which is an area calculation:

E [X] = \int x f (x) d x, operatorname {V a r} (X) = \int (x - μ)^{2} f (x) d x = E [X^{2}] - μ^{2}

Everything proved about expectation and variance in the earlier lessons carries over unchanged, because the proofs used only linearity and the fact that probabilities sum to one, both of which survive the replacement of sums by areas. Linearity still holds with no independence, variances still add for independent variables, Chebyshev's inequality still applies, and standardising still gives mean $0$ and variance $1$ .

For the uniform distribution on $[a, b]$ the mean is $(a + b) / 2$ by symmetry, and the variance works out to

operatorname {V a r} (X) = \frac{(b - a)^{2}}{12}

The $12$ is not guessable and the shape of the answer is: the variance depends only on the width, since shifting the interval cannot change the spread, and it scales as the square of the width, as the scaling rule requires. A uniform on $[0, 1]$ therefore has variance $1 / 12 \approx 0.0833$ and standard deviation $0.2887$ , which is well under the half-width of $0.5$ , as it must be, since the standard deviation is an average distance and $0.5$ is the maximum.

Example. A variable is uniform on $[2, 8]$ . Find its mean, variance and standard deviation.

Mean $(2 + 8) / 2 = 5$ . Variance $(8 - 2)^{2} / 12 = 36 / 12 = 3$ . Standard deviation $\sqrt{3} \approx 1.7321$ .

Now you. A variable is uniform on $[0, 10]$ . Find its mean and standard deviation.

Answer

Mean $5$ , variance $100 / 12 \approx 8.3333$ , standard deviation $\approx 2.8868$ .

Waiting for a Poisson event

The second continuous law needs no new ideas at all, only the previous lesson read differently.

Take a Poisson process of rate $λ$ per unit time, and let $T$ be the time until the first event. Then $T > t$ says exactly that the interval $[0, t]$ contained no events, and the count in that interval is Poisson with parameter $λ t$ , so

P (T > t) = e^{- λ t} ⟹ F (t) = 1 - e^{- λ t}

for $t \geq 0$ . This is the exponential distribution, and its density is $f (t) = λ e^{- λ t}$ , decreasing from $λ$ at zero. Its mean is $1 / λ$ , which is the only answer dimensional sense allows: a rate of $3$ events per second must give an average wait of a third of a second. Its standard deviation is also $1 / λ$ , so the wait is as spread out as it is long, and its median is $\ln 2 / λ \approx 0.6931 / λ$ , noticeably shorter than the mean because the distribution is skewed to the right.

The exponential is the continuous twin of the geometric, and it inherits memorylessness in the same form:

P (T > s + t | T > s) = \frac{e^{- λ (s + t)}}{e^{- λ s}} = e^{- λ t} = P (T > t)

Having waited $s$ already tells you nothing. A radioactive nucleus that has survived a million years is exactly as likely to decay in the next hour as a freshly made one, and this is why radioactive decay is described by a half-life at all: the half-life is the median, $\ln 2 / λ$ , and it is the same whatever the age of the sample. Carbon-14 has a half-life of $5730$ years, so its decay rate is $λ = \ln 2 / 5730$ per year and the mean lifetime of an individual atom is $1 / λ = 5730 / \ln 2 \approx 8267$ years. Mean and median differ by a factor of $1.44$ , and quoting the wrong one is a real source of error.

Memorylessness is also the exponential's biggest limitation. Anything that wears out, corrodes or fatigues has a failure rate that climbs with age, and modelling it as exponential systematically underestimates late-life failure. The test is easy to state: if the conditional chance of failing in the next hour depends on how long the item has already run, the exponential is the wrong model.

Example. A light bulb's lifetime is exponential with a mean of $1000$ hours. What is the probability that it lasts more than $1500$ hours, and what is the probability it fails within the first $500$ ?

With $λ = 1 / 1000$ , $P (T > 1500) = e^{- 1.5} \approx 0.2231$ and $P (T < 500) = 1 - e^{- 0.5} \approx 0.3935$ . Nearly $40$ percent of bulbs fail in the first half of their mean lifetime, which is the skew showing.

Now you. Buses arrive as a Poisson process at $0.5$ per hour. What is the probability of waiting more than $3$ hours for the next one?

Answer

$λ t = 0.5 \times 3 = 1.5$ , so $P (T > 3) = e^{- 1.5} \approx 0.2231$ .

Every mechanism, its own shape

The catalogue is now four laws deep. Independent trials counted give the binomial; the same trials waited for give the geometric; the rare-event limit gives the Poisson; the wait for a Poisson event gives the exponential. Each came from a mechanism, and each has a shape of its own: the binomial peaked and bounded, the geometric and exponential decaying from the left edge, the Poisson skewed at small $λ$ and less so at large.

Nothing suggests that any one shape should be more fundamental than another, and one of them is. Measured heights, weights, errors of observation and test scores fall repeatedly into a single symmetric curve that appears in none of the mechanisms above. More strangely, so do sums of things drawn from the laws above: add enough exponentials, or enough dice, or enough Bernoulli trials, and the resulting distribution converges on that same curve regardless of what went in.

The curve, and the calculations it makes routine, is the next lesson. The reason it is unavoidable is the lesson after that.