Bayesian decision theory

Subset of statistical decision theory. We have a loss function $L:\Theta \times \mathcal{A}\to {\mathbb{R}}_{\ge 0}$ where $\Theta$ is some parameter space (each $\theta$ corresponds to some distribution $P_{\theta }$) and $\mathcal{A}$ is an action set (often $\mathcal{A}=\Theta$). We have a prior $\pi$ over $\Theta$, as one does in Bayesian statistics.

The Bayes risk of a rule $\delta =\delta (X)$ which takes an action given data is

$$B_{\pi }(\delta )={\mathbb{E}}_{\theta \sim \pi }{\mathbb{E}}_{X\sim P_{\theta }}L(\theta ,\delta (X)).$$

We call ${\delta }^{\ast }$ a Bayes estimator if it minimizes the Bayes risk, i.e., $B_{\pi }({\delta }^{\ast })={\inf }_{\delta }{B}_{\pi }(\delta )$.

Another view on optimality and the Bayes estimator is via the posterior expected loss. Given data $X$, take the action which minimizes expected loss under the posterior:

$${\delta }_{\pi }(X)={\text{argmin}}_{a\in \mathcal{A}}\int L(\theta ,a)\pi (\theta |X)\text{d}\theta .$$

Then ${\delta }_{\pi }$ is the Bayes estimator. This is a huge result in Bayesian decision theory, and can be traced back to Wald, Blackwell, and Lehmann and Casella.

The proof is straightforward. Let $\ell (\delta (X)|X)=\int L(\theta ,\delta (X))\pi (\theta |X)\text{d}\theta$ be the expected posterior loss when playing $\delta (X)$. Then write

$$\begin{array}{rl}{B}_{\pi }(\theta )& ={\int }_{\Theta }{\int }_{\mathcal{X}}L(\theta ,\delta (x))p(x|\theta )\pi (\theta )\text{d}\theta \\ & ={\int }_{\Theta }{\int }_{\mathcal{X}}L(\theta ,\delta (x))\pi (\theta |x)p(x)\text{d}\theta \\ & ={\int }_{\mathcal{X}}\ell (\delta (x)|x)p(x)\text{d}x,\end{array}$$

by Fubini’s theorem. If we minimize $\ell (\delta (x)|x)$ for each $x$ (which is what minimizing the posterior loss does), then we minimize $B_{\pi }(\theta )$.