Neyman-Pearson paradigm with losses

The Neyman-Pearson paradigm is formulated in terms of Type I and Type II errors. Suppose we are testing the null $H_0$ vs the alternative $H_1$ (see hypothesis testing). In particular, we focus on constructing hypothesis tests $\varphi$ such that

$${\mathbb{P}}_{H_0}(\varphi (X^n)=1)\le \alpha ,$$

for some pre-specified $\alpha$. Here $X^n=(X_1,\dots ,X_n)$ are the observations. (Note the pre-specification is important; see issues with p-values. This motivates post-hoc hypothesis testing). Subject to this constraint we want to maximize the power of the test, i.e.,

$$\beta ={\mathbb{P}}_{H_1}(\varphi (X^n)=1).$$

Wald was the first (I think) to formulate NP in terms of decision-theory. Introduce a loss function $\ell$ with two parameters, $\ell (a,b)$, where $a\in \{0,1\}$ represents the null and alternative hypothesis, and $b\in \{0,1\}$ represents the action (accept or reject). Presumably one has $\ell (0,0)=\ell (1,1)=0$.

We restate the type-I error guarantee as a Type-I risk guarantee:

$${\mathbb{E}}_{H_0}[\ell (0,\varphi (X^n))]\le 1.$$

Note that ${\mathbb{E}}_{H_0}[\ell (0,\varphi (X^n))]={\mathbb{P}}_{H_0}(\varphi (X^n)=1)\ell (0,1)$ so in order to recapture the type-I error guarantee, we can take $\ell (0,1)=1/\alpha$, in which case

$${\mathbb{E}}_{H_0}\ell (0,\varphi (X^n))\le 1\iff {\mathbb{P}}_{H_0}(\varphi (X^n)=1)\le \alpha .$$

To recover the notion of power, introduce type-II risk, which is ${\mathbb{E}}_{H_1}[\ell (1,\varphi (X^n))].$ Write

$${\mathbb{E}}_{H_1}[\ell (1,\varphi (X^n))]=(1-\beta )\ell (0,1),$$

which relates type-II risk to power. We want to minimize type-II risk.

Using losses allows us to generalize the NP paradigm beyond binary decisions (accept/reject) and to consider more general decision spaces. Eg we can consider $\ell (a,b)$ for $b\in \mathcal{B}$. This enables post-hoc hypothesis testing, as Grunwald studies.

References

• Contributions to the theory of statistical estimation and testing hypotheses, Wald 1936.
• Beyond Neyman-Pearson, Grunwald 2024.