Estimator

An estimator is a procedure, not a number: before observing, \(\hat\theta=T(X_1,\dots,X_n)\) is a random variable with its own distribution. Applying it to the actual data gives the estimate, which is a number. Thus \(\bar X\) is an estimator of \(\mu\) and \(\bar x=7.2\) is an estimate.

To judge an estimator we ask about its bias, \(E[\hat\theta]-\theta\), and its variability, \(\operatorname{Var}(\hat\theta)\). It is unbiased if the bias is zero: on average it hits the target. Both ideas are summarized in the mean squared error, \(\mathrm{MSE}=\operatorname{Var}(\hat\theta)+\text{bias}^2\), which sometimes favors a slightly biased but much more stable estimator.

Guiding example: \(\bar X\) is unbiased for \(\mu\) and the sample variance \(S^2=\tfrac{1}{n-1}\sum(X_i-\bar X)^2\) is unbiased for \(\sigma^2\); dividing by \(n\) instead of \(n-1\) would bias it downward. An estimator is consistent if it approaches the parameter as \(n\) grows, which the law of large numbers guarantees for \(\bar X\).

A typical mistake is to think an unbiased estimator is always the best: it can have a huge variance, like \(X_1\) compared with \(\bar X\). Another is forgetting that the estimate changes from sample to sample; quantifying that uncertainty is exactly the job of confidence intervals and hypothesis tests.