Conditional probability

Every probability computed so far assumed the whole sample space was still in play, and information destroys that assumption. This lesson derives the one rule that says what a probability becomes once something is known, and then uses it to define independence, to build trees, and to settle a game show argument that split a magazine's readership in 1990.

Renormalising onto what is left

Two fair dice are rolled and you are told the total is at least nine. What is the probability that at least one die shows a six?

The original sample space had $36$ equally likely pairs. The news has killed every pair whose total is eight or less, leaving $10$ pairs, and those $10$ are still equally likely relative to one another, because nothing distinguished them before and the news treated them alike. Of the survivors, $7$ contain a six. So the answer is $7 / 10 = 0.7$ , against an unconditional $11 / 36 \approx 0.306$ .

Two things happened. The sample space shrank to the event that was reported, and the probabilities of the survivors were scaled up so they again sum to one. That scaling is a division by the probability of the reported event, and writing it in general gives the definition. For events $A$ and $B$ with $P (B) > 0$ ,

P (A | B) = \frac{P (A \cap B)}{P (B)}

read "the probability of $A$ given $B$ ". The numerator keeps only the part of $A$ that survives, and the denominator is the renormalisation. Check it on the dice: $P (A \cap B) = 7 / 36$ , $P (B) = 10 / 36$ , and the ratio is $7 / 10$ .

This is a definition rather than a theorem, but it is not arbitrary. For any fixed $B$ , the assignment $A \mapsto P (A | B)$ satisfies all three of Kolmogorov's axioms: it is non-negative, it gives $Ω$ probability $P (B) / P (B) = 1$ , and it is additive because the numerator is. So a conditional probability is a probability, and every rule from the earlier lesson holds inside it, including $P (A^{c} | B) = 1 - P (A | B)$ . What is not true, and is a standard error, is that anything holds when you condition on different things: $P (A | B) + P (A | B^{c})$ is not $1$ and need not be anything in particular.

The chain rule

Multiply the definition out and it becomes a way to build probabilities rather than to update them:

P (A \cap B) = P (B) P (A | B)

and iterating, for three events,

P (A \cap B \cap C) = P (A) P (B | A) P (C | A \cap B)

This is how sequential experiments are actually computed. Draw two cards from a full deck: the first is an ace with probability $4 / 52$ , and given that, the second is an ace with probability $3 / 51$ , since one ace and one card are gone. So

P (two aces) = \frac{4}{52} \times \frac{3}{51} = \frac{12}{2652} \approx 0.004525

Three aces adds a factor $2 / 50$ and gives $\approx 0.000181$ . The counting lesson would reach the same answers through $C (4, 2) / C (52, 2)$ , and the chain rule is usually quicker because each factor is a small local question.

The same pattern handles any sequential draw. An urn holding $5$ red and $3$ black balls gives up two reds in a row with probability $(5 / 8) (4 / 7) = 20 / 56 \approx 0.3571$ , and two blacks with probability $(3 / 8) (2 / 7) = 6 / 56 \approx 0.1071$ , each factor asking only what is left in the urn at that moment.

Independence, defined rather than felt

Sometimes the news changes nothing: $P (A | B) = P (A)$ . Substituting into the chain rule turns this into a symmetric statement with no division in it, and that is the form to take as the definition. Events $A$ and $B$ are independent when

P (A \cap B) = P (A) P (B)

The symmetry is worth noticing. If $B$ tells you nothing about $A$ then $A$ tells you nothing about $B$ , which is not obvious from the conditional form but is immediate from this one. The definition also handles the awkward case $P (B) = 0$ , where the conditional is undefined but the product form is fine.

Independent is not the same as disjoint, and confusing them is the commonest slip in the subject. Disjoint events are maximally dependent: if $A$ and $B$ cannot both happen, then learning $B$ occurred tells you $A$ certainly did not, so $P (A | B) = 0$ . Two disjoint events of positive probability are never independent.

Independence is also not settled by checking pairs. Roll two fair dice and take $A$ as "first die is even", $B$ as "second die is even", $C$ as "the total is even". Each has probability $1 / 2$ , and each pair is independent: knowing the first die is even says nothing about the second, and knowing the second die is even leaves the total even exactly when the first is, still probability $1 / 2$ . But the three together are not independent, because any two of them force the third, so $P (A \cap B \cap C) = P (A \cap B) = 1 / 4$ , not $(1 / 2)^{3} = 1 / 8$ . Full independence of a family means every subfamily multiplies, not merely every pair.

In practice independence is an assumption imported from the physics of the situation, not something read off the numbers. Successive coin tosses are independent because a coin has no memory. Successive days of rain are not. Assuming independence where it does not hold is how risk models fail, since it makes simultaneous disasters look astronomically unlikely when they are merely unlikely.

Example. A component fails during a mission with probability $0.02$ , independently of the other components. A system contains four such components and works only if all four survive. What is the probability that the system works?

Independence lets the probabilities multiply: $P (all survive) = {0.98}^{4} \approx 0.9224$ , so the system fails with probability about $0.0776$ , nearly four times the single-component risk.

Now you. The same system is built with six such components instead of four. What is the probability that it works?

Answer

${0.98}^{6} \approx 0.8858$ , so the failure probability rises to about $0.1142$ .

The law of total probability

Conditioning is most useful when the conditioning events form a partition: a set of alternatives $B_{1}, B_{2}, \dots, B_{n}$ , exactly one of which occurs. Any event $A$ is then chopped into the disjoint pieces $A \cap B_{i}$ , and additivity plus the chain rule gives

P (A) = \sum_{i} P (B_{i}) P (A | B_{i})

the law of total probability. Read it as an instruction: split the world into cases, work out the answer within each case, and average the answers weighted by how likely each case is.

Take a factory with three machines. Machine one makes $50$ percent of the output with a defect rate of $1$ percent, machine two makes $30$ percent at $2$ percent, machine three makes $20$ percent at $3$ percent. An item is picked at random from the day's production. Then

P (defective) = 0.5 (0.01) + 0.3 (0.02) + 0.2 (0.03) = 0.005 + 0.006 + 0.006 = 0.017

so $1.7$ percent overall, which lies between the best and worst machine rates and is nearer the best, because the best machine makes the most. Note what the calculation does not answer: given that an item is defective, which machine most likely made it? Machine one contributes the largest single share of the defects, $0.005$ out of $0.017$ , or $29.4$ percent, despite having the lowest defect rate, simply because it makes half of everything. That reversal is the subject of the next lesson.

Example. From the urn of $5$ red and $3$ black balls, two are drawn without replacement. What is the probability that the second is red?

Condition on the first. If the first was red (probability $5 / 8$ ), four of the remaining seven are red. If it was black (probability $3 / 8$ ), five of seven are. So

P (second red) = \frac{5}{8} \cdot \frac{4}{7} + \frac{3}{8} \cdot \frac{5}{7} = \frac{20 + 15}{56} = \frac{35}{56} = 0.625

which is exactly $5 / 8$ , the probability the first is red. This is not a coincidence: before you look at anything, the ball in position two is as likely to be red as the ball in position one, because the balls were never ordered by colour. Symmetry gives the answer in one line where conditioning takes three.

Now you. A box holds $4$ white and $6$ green counters. Two are drawn without replacement. Use the law of total probability to find the probability that the second is white, and check it against the symmetry argument.

Answer

$\frac{4}{10} \cdot \frac{3}{9} + \frac{6}{10} \cdot \frac{4}{9} = \frac{12 + 24}{90} = \frac{36}{90} = 0.4$ , which is $4 / 10$ as symmetry predicts.

The three doors

In September 1990 Marilyn vos Savant answered a reader's question in Parade magazine and received thousands of letters telling her she was wrong, many from people with doctorates. The setup: three doors, a car behind one and goats behind the other two. You pick a door. The host, who knows where the car is, opens one of the other two to reveal a goat, and always can. He offers you the switch. Should you take it?

The answer is yes, and switching wins with probability $2 / 3$ . The clean way to see it is to condition on your first pick, which is the only random thing you control. With probability $1 / 3$ your first pick was the car, in which case switching loses. With probability $2 / 3$ it was a goat, in which case the host is forced to reveal the other goat, so the remaining door holds the car and switching wins. There are no other cases, so switching wins with probability $2 / 3$ and staying with probability $1 / 3$ .

The reason the wrong answer is so seductive is that the host's action feels like it leaves two equally likely doors. It does not, because the host's choice is not random: he never opens the car, and he never opens your door. Those two constraints leak information about where the car is, and the leak flows entirely to the door he did not open. Change the rules so the host opens a door at random and happens to reveal a goat, and the two remaining doors really are equally likely, and switching gains nothing. The puzzle is a lesson about conditioning on what actually happened, including on the rule that generated it.

Example. The same game with $100$ doors: you pick one, and the host opens $98$ others, all goats, always able to. What is the probability that switching wins?

Your first pick is the car with probability $1 / 100$ . Otherwise, with probability $99 / 100$ , the single door the host left closed is the car, since he was forced to leave it. Switching wins with probability $0.99$ .

Now you. The game with $5$ doors: you pick one, the host opens $3$ others, all goats. What is the probability that switching to the one remaining door wins?

Answer

Your pick is right with probability $1 / 5$ , so switching wins with probability $4 / 5 = 0.8$ .

The direction of the arrow

Conditioning has given a definition, a chain rule for building sequences, a real test for independence, and a way to average over cases. Every one of those runs the same way: from a cause, or an earlier stage, to what is observed. The factory calculation went from machine to defect. The urn went from first draw to second. The doors went from where the car was to what the host did.

Real questions run backwards. The defect is on the bench and the machine is unknown. The test result is positive and the disease is unknown. The evidence is in the courtroom and the guilt is unknown. What is wanted is $P (cause | observation)$ , and what is easy to state is $P (observation | cause)$ , since the second is a property of the mechanism and the first is not.

Nothing new is needed to reverse them, because the definition of conditional probability is already symmetric in a way that has not been exploited: $P (A \cap B)$ can be expanded in two orders. Setting the two expansions equal gives the most consequential single formula in the subject, and it is the next lesson.