Variance and spread

Expectation compresses a distribution to a single number and cannot tell a certainty from a lottery with the same average. This lesson builds the second number, the one that measures spread, and finds that it has a property expectation does not share and a consequence expectation cannot deliver.

Choosing a measure of spread

Spread means typical distance from the centre, so the obvious candidate is the mean of $X - μ$ , writing $μ = E [X]$ throughout. That candidate dies immediately: $E [X - μ] = E [X] - μ = 0$ for every variable whatsoever, since the deviations above and below the mean cancel exactly. Any usable measure has to remove the signs first.

Two ways to do it. Take absolute values, giving the mean absolute deviation $E | X - μ |$ , or take squares, giving $E [(X - μ)^{2}]$ . The absolute version is the more natural reading of "typical distance" and is almost never used, for reasons that are practical rather than deep. The absolute value has no derivative at zero and does not expand when a sum is substituted into it, so a formula containing it usually stays stuck. The square expands, and it is that expansion, three lines below, that produces the additivity the whole subject runs on.

So define the variance

operatorname {V a r} (X) = E [(X - μ)^{2}]

and, because squaring changed the units, the standard deviation $σ = \sqrt{operatorname {V a r} (X)}$ , which is back in the units of $X$ and is the number to quote. A variance is never negative, since it averages squares, and it is zero exactly when $X$ equals $μ$ with probability one.

Expanding the square gives a version that is easier to compute:

operatorname {V a r} (X) = E [X^{2} - 2 μ X + μ^{2}] = E [X^{2}] - 2 μ E [X] + μ^{2} = E [X^{2}] - μ^{2}

so the variance is the mean of the square minus the square of the mean, and the gap noticed in the previous lesson between $E [X^{2}] = 91 / 6$ and $(E [X])^{2} = 12.25$ for a fair die is exactly its variance: $35 / 12 \approx 2.9167$ , with $σ \approx 1.7078$ .

What scaling does

Shifting a variable moves its mean and leaves its spread alone; stretching it stretches the spread. Both facts follow from the definition. For constants $a$ and $b$ , the variable $a X + b$ has mean $a μ + b$ , so its deviation from its own mean is $a (X - μ)$ , and squaring introduces $a^{2}$ :

operatorname {V a r} (a X + b) = a^{2} operatorname {V a r} (X), σ_{a X + b} = | a | σ_{X}

The additive constant vanishes entirely, which is right: adding ten to every value moves the whole distribution without spreading it. The factor $a^{2}$ rather than $a$ is the price of working with squares, and the standard deviation, taking the square root, scales the sensible way.

This licenses standardisation. Given any $X$ with mean $μ$ and standard deviation $σ > 0$ , the variable

Z = \frac{X - μ}{σ}

has mean $0$ and variance $1$ , since the shift kills the mean and the division by $σ$ divides the variance by $σ^{2}$ . A value of $Z$ counts how many standard deviations $X$ is from its mean, and it is dimensionless, so heights and test scores and share prices become comparable. This construction returns in the lesson on the normal curve, where it does most of the work.

A quick check on the rule: a game paying $10 X + 5$ units on a roll of a fair die has mean $10 (3.5) + 5 = 40$ units and standard deviation $10 \times 1.7078 = 17.078$ units, the additive $5$ contributing nothing to the spread and everything to the centre.

Why variances add

Expectation adds unconditionally. Variance does not, and the exact condition is worth deriving. Let $X$ and $Y$ have means $μ_{X}$ and $μ_{Y}$ . Then

operatorname {V a r} (X + Y) = E [((X - μ_{X}) + (Y - μ_{Y}))^{2}] = operatorname {V a r} (X) + operatorname {V a r} (Y) + 2 E [(X - μ_{X}) (Y - μ_{Y})]

by expanding the square inside the expectation and using linearity on the three resulting terms. The last piece is the covariance

operatorname {C o v} (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] = E [X Y] - E [X] E [Y]

which is positive when the two tend to be above their means together, negative when one being high goes with the other being low, and zero when there is no such tendency on average.

If $X$ and $Y$ are independent then $E [X Y] = E [X] E [Y]$ , because the joint mass function factorises and the double sum splits into a product of two sums. So independence forces the covariance to zero and leaves

operatorname {V a r} (X + Y) = operatorname {V a r} (X) + operatorname {V a r} (Y)

Two cautions. First, the implication runs one way only: zero covariance does not imply independence. Let $X$ be $- 1$ , $0$ or $1$ with probability $1 / 3$ each and $Y = X^{2}$ . Then $E [X Y] = E [X^{3}] = 0 = E [X] E [Y]$ , so the covariance vanishes, yet $Y$ is a function of $X$ and could hardly be more dependent. Covariance detects linear association and is blind to the rest.

Second, dependence can cancel spread entirely. Let $Y = 7 - X$ for a die roll $X$ . Both have variance $35 / 12$ , and their sum is the constant $7$ , whose variance is zero. Here the covariance is $- 35 / 12$ and exactly cancels the two positive terms. Assuming variances add when the variables are linked is how a portfolio of correlated assets comes to look far safer than it is.

Example. Two fair dice are rolled independently. What are the variance and standard deviation of the total?

Each die has variance $35 / 12$ , and independence lets them add: $operatorname {V a r} = 70 / 12 \approx 5.8333$ , so $σ \approx 2.4152$ . The mean is $7$ , so a total is typically about two and a half away from seven.

Now you. Three independent fair dice are rolled. Find the variance and standard deviation of the total.

Answer

$operatorname {V a r} = 3 \times 35 / 12 = 105 / 12 = 8.75$ , so $σ = \sqrt{8.75} \approx 2.9580$ , with mean $10.5$ .

The square root of n

Now take $n$ independent copies of the same variable, each with mean $μ$ and variance $σ^{2}$ : repeated measurements, repeated bets, repeated trials. Their sum $S_{n}$ has mean $n μ$ by linearity and variance $n σ^{2}$ by additivity, so

σ_{S_{n}} = σ \sqrt{n}

The sum's spread grows, but only as the square root of the number of terms, while its mean grows as $n$ itself. Divide through to get the average ${\overline{X}}_{n} = S_{n} / n$ , which by the scaling rule has variance $n σ^{2} / n^{2} = σ^{2} / n$ and therefore

σ_{{\overline{X}}_{n}} = \frac{σ}{\sqrt{n}}

This single formula is why measurements are repeated. Averaging four readings halves the spread, twenty-five readings divides it by five, ten thousand readings by a hundred. It is also a warning about cost: each further factor of two in precision costs four times the data, so precision gets expensive fast.

It also explains the two columns of Kerrich's coin table from the first lesson. The number of heads in $n$ tosses has standard deviation $\sqrt{n} / 2$ , which grows without bound, so the absolute surplus of heads drifts further from zero. The fraction of heads has standard deviation $1 / (2 \sqrt{n})$ , which shrinks to nothing. Both columns behave exactly as they must, and neither is evidence about the coin.

Example. A measurement has standard deviation $0.8$ grams. How many independent repeats are needed for the average to have a standard deviation of $0.1$ grams?

Set $0.8 / \sqrt{n} = 0.1$ , so $\sqrt{n} = 8$ and $n = 64$ measurements. Halving that target again, to $0.05$ grams, would take $256$ .

Now you. An opinion poll of $1000$ people estimates a proportion whose standard deviation is at most $0.5 / \sqrt{n}$ . What is that bound for $n = 1000$ , and how many people would be needed to halve it?

Answer

$0.5 / \sqrt{1000} \approx 0.0158$ , about $1.6$ percentage points. Halving it needs four times the sample: $4000$ people.

Chebyshev's guarantee

Variance is more than a descriptive number: it bounds how much probability can sit far from the mean, for any distribution at all. Take any $k > 0$ and let $A$ be the event $| X - μ | \geq k σ$ . Then

σ^{2} = E [(X - μ)^{2}] \geq E [(X - μ)^{2} I_{A}] \geq k^{2} σ^{2} P (A)

The first inequality drops the contribution from outcomes outside $A$ , which is non-negative. The second replaces $(X - μ)^{2}$ by its minimum value on $A$ , which is $k^{2} σ^{2}$ by the definition of $A$ . Dividing by $k^{2} σ^{2}$ gives Chebyshev's inequality, published by Pafnuty Chebyshev in 1867:

P (| X - μ | \geq k σ) \leq \frac{1}{k^{2}}

No distribution can put more than a quarter of its probability two or more standard deviations from its mean, and never more than a ninth beyond three, whatever shape it has: no symmetry assumed, no bell curve, nothing but a finite variance.

The bound is very loose in practice, and deliberately so, since it must cover the worst case. Toss a fair coin $100$ times: the number of heads has mean $50$ and standard deviation $5$ , so Chebyshev caps $P (| X - 50 | \geq 15)$ at $1 / 9 \approx 0.1111$ . The true value, computable exactly from the binomial formula of the next lesson, is $0.00352$ , smaller by a factor of thirty. Chebyshev is not for computing probabilities. It is for proving that they must be small, which is exactly what is needed to prove the law of large numbers later in this course.

Example. A production line fills bags with mean $500$ grams and standard deviation $8$ grams. What does Chebyshev guarantee about the fraction of bags between $480$ and $520$ grams?

The interval is $μ \pm 20$ grams, which is $k = 20 / 8 = 2.5$ standard deviations. Chebyshev bounds the outside at $1 / {2.5}^{2} = 0.16$ , so at least $84$ percent of bags lie inside. If the weights happen to follow a normal curve the real figure is above $98$ percent, and the guarantee holds whether or not they do.

Now you. For the same line, what fraction of bags does Chebyshev guarantee lie between $476$ and $524$ grams?

Answer

$k = 24 / 8 = 3$ , so at most $1 / 9 \approx 0.1111$ lies outside and at least $88.9$ percent lies inside.

What two numbers still miss

Mean and variance now exist for any variable with a finite second moment, they behave predictably under scaling, they add under independence, and they yield a universal tail bound. That is enough to say something about every distribution and not enough to say much about any particular one.

Two distributions can share a mean and a variance and look nothing alike. Symmetric and skewed, single-peaked and twin-peaked, bounded and unbounded: none of that is visible in the first two moments. Worse, some distributions have no finite variance at all, and for those Chebyshev says nothing and the square root of $n$ law fails outright. Such heavy-tailed cases are not exotic curiosities; they appear in insurance losses and in financial returns, where a model chosen for its convenient variance quietly understates the risk of the events that matter.

The productive move is to stop treating distributions one at a time. A handful of mechanisms generate most of the distributions that occur, and each mechanism fixes the whole mass function, with the mean and variance falling out of it. The first and most important of those mechanisms is repeated independent trials with a fixed success probability, and it is the next lesson.