Law of large numbers

Let \(X_1,X_2,\dots\) be independent, with the same expectation \(\mu\) and finite variance \(\sigma^2\). The (weak) law says that, for every \(\varepsilon>0\), \(P(|\bar X_n-\mu|\ge\varepsilon)\to0\) as \(n\to\infty\). The strong form says more: with probability 1, the sequence of averages converges to \(\mu\).

With finite variance the proof is a direct application of Chebyshev: since \(E[\bar X_n]=\mu\) and \(\operatorname{Var}(\bar X_n)=\sigma^2/n\), we get \(P(|\bar X_n-\mu|\ge\varepsilon)\le\sigma^2/(n\varepsilon^2)\), which tends to 0. It also gives an explicit bound: to guarantee an error below \(\varepsilon\) with failure probability \(\delta\), it suffices that \(n\ge\sigma^2/(\delta\varepsilon^2)\).

A guiding example is the relative frequency of an event: if \(X_i=1\) when the event occurs in repetition \(i\), then \(\bar X_n\) is the proportion of times it occurs and tends to its probability. This justifies reading a probability as a long-run frequency, and also simulation and estimation methods.

Two typical errors: believing the law «compensates» past streaks (trials are independent) and thinking the absolute difference \(S_n-n\mu\) vanishes, when what stabilizes is the average. The speed and shape of the remaining fluctuation are described by the central limit theorem.