Bayes' theorem

Writing \(P(A\cap B)\) in two ways, \(P(B\mid A)P(A)=P(A\mid B)P(B)\), and solving gives \(P(B\mid A)=P(A\mid B)P(B)/P(A)\). When \(B_1,\dots,B_n\) is a partition, the denominator \(P(A)\) is computed with total probability, which yields the full formula. \(P(B_j)\) is called the prior, \(P(A\mid B_j)\) the likelihood and \(P(B_j\mid A)\) the posterior probability.

Guiding example: a disease affects 1 % of the population; the test is positive for 99 % of the sick and 5 % of the healthy. A positive result has probability \(\dfrac{0.01\cdot0.99}{0.01\cdot0.99+0.99\cdot0.05}=\tfrac16\) of belonging to a sick person. Although the test looks very reliable, the disease is so rare that false positives among the healthy outnumber true positives.

So the posterior depends both on the evidence and on the initial proportion of each case. With a coin that is either fair or two-headed, each consecutive head raises the suspicion of the trick coin: after three heads it goes from 1/2 to 8/9. This is the basis of Bayesian inference.

The most common mistake is confusing \(P(A\mid B)\) with \(P(B\mid A)\), or ignoring the prior (the «base rate fallacy»). If \(P(A\mid B_i)\) is the same for every case, observing \(A\) changes nothing: the posterior equals the prior.