Random variables and expectation

Everything so far has been about events, which either happen or do not. Most questions worth asking concern a quantity instead: how many defective items, how long a wait, how much money. This lesson attaches numbers to outcomes, defines the average of such a number, and proves one property of that average that does more work than any other result in the subject.

A number attached to every outcome

A random variable is a function from the sample space to the real numbers. It assigns to each outcome $\omega$ a value $X(\omega )$. That is all it is, and the name is doubly misleading: it is not random, being a fixed function, and it is not a variable, being a function. The randomness lives in which outcome occurs, and $X$ merely reports a number about it.

Roll two dice, so $\Omega$ is the $36$ ordered pairs, and let $X$ be the total. Then $X((3,4))=7$, and $X$ is defined on every outcome with no ambiguity. Once $X$ exists, statements about it become events in the old sense: "$X=7$" names the set of six pairs that total seven, and "$X\ge 10$" names a set of six pairs, so both have probabilities by the rules already established.

A variable taking values in a finite or countable set is discrete, which covers everything in this lesson and the next three. Its behaviour is captured completely by the probability mass function

which is non-negative and sums to $1$ over all possible values, since the events $\{X=x\}$ partition $\Omega$. For the two-dice total, the mass function runs $1/36,2/36,\dots ,6/36,\dots ,2/36,1/36$ across the values $2$ to $12$, and those eleven numbers sum to $36/36$.

The mass function throws away the sample space, and that is its point. Two completely different experiments with the same mass function are interchangeable for every purpose that mentions only $X$. From here on the underlying $\Omega$ is rarely written down.

The average of a random variable

Suppose the experiment is repeated $N$ times and $X$ takes the value $x$ on about $Np(x)$ of them. The arithmetic mean of the observed values is then

with $N$ cancelling. That expression is taken as the definition of the expectation, also called the expected value or the mean:

It is a weighted average of the possible values, weighted by their probabilities, and it is a fixed number, not a random one. Whether the long-run average really converges to it is a theorem proved much later in this course, not an assumption made here.

For a single fair die, $E[X]=(1+2+3+4+5+6)/6=3.5$, a value the die can never show. Expectation is a centre of mass, not a prediction, and reading it as "what will happen" is the first of several ways it misleads. For the two-dice total, $E[X]=7$, which the sum can show, and which sits at the centre of the symmetric mass function.

Example. A European roulette wheel has $37$ pockets: $18$ red, $18$ black and one green zero. You stake $1$ unit on red, winning $1$ if red comes up and losing your stake otherwise. What is the expected gain per spin?

so the house keeps about $2.7$ percent of everything staked, on average, whatever the bet.

Now you. An American wheel has $38$ pockets, with both a zero and a double zero, so $18$ are red. What is the expected gain per unit staked on red?

Functions of a random variable

If $X$ is a random variable then so is $g(X)$ for any function $g$, since a function of a function is a function. Its expectation could be found by first working out the mass function of $g(X)$, and it never has to be, because

The proof is a regrouping: gather the outcomes according to the value of $X$ rather than the value of $g(X)$, and every outcome is still counted exactly once with its own probability. The result is often called the law of the unconscious statistician, from the habit of using it without noticing that it needs proof.

Two consequences are worth having on hand. First, $E[aX+b]=aE[X]+b$ for constants $a$ and $b$, since the sum splits and $\sum _{x}p(x)=1$. Expectation moves through a change of units unharmed: a temperature in Fahrenheit has $9/5$ times the expected Celsius value plus $32$. Second, $E[X^2]$ is generally not $(E[X]{)}^2$. For a fair die, $E[X^2]=(1+4+9+16+25+36)/6=91/6\approx 15.1667$, while $(E[X]{)}^2=12.25$. The gap of $2.9167$ is not an error; the next lesson identifies it as the variance.

The practical consequence is that a payout must be averaged after the function is applied, never before. A stall charging $2$ units to roll a die and paying out the square of the result takes in $2$ and pays out $E[X^2]\approx 15.1667$, losing about $13.17$ units a play; pricing it from $(E[X]{)}^2=12.25$, or worse from $E[X]=3.5$, would have hidden that.

Linearity, which asks for nothing

Here is the result that earns its keep. For any random variables $X$ and $Y$ defined on the same sample space,

The proof works outcome by outcome rather than value by value. Writing the expectation as a sum over $\Omega$,

which is nothing but the associativity of addition. Notice what is absent: no independence, no assumption about how $X$ and $Y$ relate, not even that they are different quantities. It holds when $Y=X$, when $Y=-X$, when $Y$ is a deliberately hostile function of $X$. Extended to any finite number of terms it says the expectation of a sum is always the sum of the expectations.

This is stronger than it looks, because most quantities of interest are sums of simple pieces whose joint behaviour is a nightmare and whose individual behaviour is trivial. The two-dice total illustrates the pattern in miniature: rather than summing $\sum _{x}xp(x)$ over the eleven values with their awkward weights, write $X=X_1+X_2$ and get $3.5+3.5=7$ immediately.

Example. Five cards are dealt from a shuffled deck. What is the expected number of hearts?

Let $X_i$ be $1$ if the card in position $i$ is a heart and $0$ otherwise, so the number of hearts is $X_1+\cdots +X_5$. Each individual card is equally likely to be any of the $52$, so $E[X_i]=P(\text{heart})=13/52=0.25$. Linearity gives $5\times 0.25=1.25$ hearts expected. The cards are emphatically not independent, since dealing a heart makes the next heart less likely, and linearity does not care.

Now you. Ten cards are dealt from the same deck. What is the expected number of aces?

Indicators, and the problems they crack

The technique in that example deserves a name. An indicator variable $I_A$ takes the value $1$ when the event $A$ occurs and $0$ when it does not, and its expectation is

so expectation and probability are the same operation on indicators. Any count is a sum of indicators, one per thing that might be counted, and by linearity its expectation is the sum of the separate probabilities. Nothing about dependence ever enters.

Take the classic: $n$ letters are put at random into $n$ addressed envelopes. How many arrive at the right address? The exact distribution of the number of matches is a genuinely awkward object involving alternating sums. Its mean is not. Let $I_i$ indicate that letter $i$ is in its own envelope. Any given letter is equally likely to be in any of the $n$ envelopes, so $E[I_i]=1/n$, and the expected number of matches is $n\times (1/n)=1$ for every $n$. One letter arrives correctly on average, whether there are ten letters or ten million.

Or the coupon collector. There are $n$ different coupons, one at random in each packet. Let $T_k$ be the number of packets bought while holding exactly $k$ distinct coupons. Each purchase in that stretch is new with probability $(n-k)/n$, and a wait for a success with constant probability $p$ has expectation $1/p$, proved in the lesson on the binomial family. So $E[T_k]=n/(n-k)$, and the total is

For $n=6$, a full set of six needs $6\times 2.45=14.7$ packets on average, and for $n=50$ it is $225$. Collecting the last few is most of the work: the final coupon alone takes $n$ packets on average.

Example. A fair die is rolled six times. What is the expected number of distinct faces seen?

Let $I_f$ indicate that face $f$ appears at least once. It fails to appear with probability $(5/6{)}^6\approx 0.3349$, so $E[I_f]\approx 0.6651$. Summing over the six faces gives $6\times 0.6651\approx 3.99$ distinct faces, about four.

Now you. A fair coin is tossed $10$ times. What is the expected number of tosses that match the toss before them, out of the $9$ adjacent pairs?

Where the mean is not enough

Expectation compresses a whole distribution into one number, and the compression discards things that matter.

Consider two propositions. The first pays $1$ unit with certainty. The second pays $1000000$ units with probability ${10}^{-6}$ and nothing otherwise. Both have expectation exactly $1$, and nobody treats them as equivalent. Expectation cannot distinguish a certainty from a lottery, because it never asks how far the outcomes stray from their average.

The extreme case was posed by Nicolas Bernoulli in 1713. A coin is tossed until it lands heads; if this takes $n$ tosses, the payout is $2^n$ units. The probability of stopping at toss $n$ is $2^{-n}$, so each term of the expectation contributes $2^{-n}\times 2^n=1$, and the sum over infinitely many terms diverges. The expected payout is infinite, and no one will pay even $100$ units to play, since a payout above $100$ requires at least seven tosses, which happens with probability $2^{-6}=0.015625$, about once in $64$ plays. Daniel Bernoulli's 1738 resolution, that people value money by something growing more slowly than the amount, launched utility theory, but the plain mathematical point stands on its own: an expectation can exist and be useless, or fail to exist while the variable is perfectly well behaved.

What is missing is a measure of how widely the values are spread around the mean. Building one, discovering that it adds when expectations of products do not, and extracting a guarantee about how far from the mean a variable can stray, is the next lesson.