statistical decision theory

You’re doing some statistical inference on a Friday night, as one does. You compute some estimate $\hat{\theta }$ of the true parameter $\theta$. But how do you know if it’s any good?

You could look at whether it’s consistent, or whether it’s unbiased, or whether $\hat{\theta }-\theta$, obeys a CLT around the true parameter. But who’s to say? The point of statistical decision theory is to provide a formal framework for answering this question.

One begins with a loss function $L:\Theta \times \Theta \to {\mathbb{R}}_{\ge 0}$ which measures how good the estimator is. (eg $L(\theta ,\varphi )=(\theta -\varphi {)}^2$ is the ol’ classic squared error. There’s also ${\ell }_1$ error, $L_p$ loss, zero-one loss, etc.) The risk of an estimator $\hat{\theta }(X)$ the expected loss, i.e.,

$$R(\theta ,\hat{\theta })={\mathbb{E}}_{X\sim P_{\theta }}R(\theta ,\hat{\theta }(X)).$$

Note that $\theta$ is fixed; the expectation is with respect to the randomness of the data.

It’s tempting to compare various estimators, ${\hat{\theta }}_1$, ${\hat{\theta }}_2$ by comparing their risks. The one with the lower risk wins, right? But ${\hat{\theta }}_1$ might have lower risk than ${\hat{\theta }}_2$ on some parameters $\theta$ and higher risk on others (in fact, this is almost certainly the case for any reasonable estimators). In that case how do we say which one is better?

The two most common strategies are to consider the maximum risk and the Bayesian risk.

• The maximum risk considers the worst case over all parameters $\theta$, $M(\hat{\theta })={\sup }_{\theta }R(\theta ,\hat{\theta })$. The estimator minimizing the maximum risk is the minimax estimator. A huge swath of modern statistics is taken up with the question of determining minimax rates and finding minimax estimators.
• The Bayesian risk puts a prior $\pi$ over $\Theta$ and then considers the Bayes risk: $B(\hat{\theta })=\int R(\theta ,\hat{\theta })\pi (\theta )\text{d}\theta$. There’s lots more to say here, see Bayesian decision theory.