Neyman-Pearson lemma

States that a variant of the likelihood-ratio test is the uniformly most powerful test among all level-$\alpha$ tests when testing simple nulls against simple alternatives. The NP lemma is generalized to monotone likelihood ratio families by the Karlin-Rubin theorem.

Suppose we are testing $H_0:\theta ={\theta }_0$ vs $H_1:\theta ={\theta }_1$. Consider the likelihood ratio test, which says to reject if $p_1({\theta }_1)/p_0({\theta }_0)\ge k$ where $k$ is chosen so that

$${\mathbb{P}}_0(p_1({\theta }_1)/p_0({\theta }_0)\ge k)=\alpha .$$

where ${\mathbb{P}}_0$/${\mathbb{P}}_1$ are the distributions under the null/alternative and $p_0/p_1$ are their densities. The NP lemma states that this test is UMP.

The NP test is often written as

$$\varphi (X)=\{\begin{array}{ll}1,& L(X)>c,\\ \gamma ,& L(X)=c,\\ 0,& L(X)<c,\end{array}$$

where $L$ is the likelihood ratio $p_1/p_0$ and $\gamma$ and $c$ are chosen that such that ${\mathbb{E}}_{P_0}[\varphi (X)]=\alpha$. In the continuous case, $\gamma$ is unimportant since $L(X)=c$ has probability zero. But it matters in the discrete case (Neyman-Pearson lemma for discrete distributions) where it’s used to ensure that the test has size exactly $\alpha$.

Proof 1

Let $\varphi$ denote the likelihood ratio test and $\psi$ denote any other level $\alpha$ test. Consider the integral

$$\int (\varphi (x)-\psi (x))(p_1(x)-k{p}_0(x))\text{d}x.$$

We claim this integral is lower bounded by zero. If $\varphi (x)=\psi (x)$ then this is immediate. If $\psi (x)=1$ then $p_1(x)\ge k{p}_0(x)$ by definition, so the product of both terms in the integral is nonnegative. Likewise, if $\psi (x)=0$ then both terms are non-positive. The claim follows.

This gives that

$$\begin{array}{rl}\int (\varphi (x)-\psi (x))p_1(x)\text{d}x& \ge k\int (\varphi (x)-\psi (x))p_0(x)\text{d}x\\ & =k{\mathbb{E}}_{H_0}[\varphi (X)-\psi (X)]\\ & =k({\mathbb{P}}_{H_0}(\varphi (X)=1)-{\mathbb{P}}_{H_0}(\psi (X)=1))\\ & \ge k\alpha -k\alpha =0,\end{array}$$

since $\varphi$ is size $\alpha$ and $\psi$ is level $\alpha$, by assumption. Therefore,

$${\mathbb{P}}_{H_1}(\varphi (X)=1)-{\mathbb{P}}_{H_1}(\psi (X)=1)\ge 0,$$

which is the desired result.

Proof 2

Turn the problem into a constrained optimization problem using Lagrange multipliers. We want to maximize $\int \varphi (x)p_1(x)\text{d}x$ subject to the constraint that $\int \varphi (x)p_0(x)\text{d}x\le \alpha$. So we want to maximize

$$\begin{array}{rl}\mathcal{L}(\varphi ,\lambda )& =\int \varphi (x)p_1(x)\text{d}x-\lambda \left(\int \varphi (x)p_0(x)-\alpha \right)\\ & =\int \varphi (x)(p_1(x)-\lambda p_0(x))\text{d}x-\lambda \alpha .\end{array}$$

For a given $\lambda$, the test $\varphi$ which maximizes this clearly is 1 when $p_1(x)\ge \lambda p_0(x)$ and 0 otherwise. And what is $\lambda$? Given our choice of $\varphi$ we have

$$\int \varphi (x)p_0(x)={\int }_{p_1(x)\ge \lambda p_0(x)}{p}_0(x)\text{d}x={\mathbb{P}}_0(p_1(x)/p_0(x)\ge \lambda ).$$

If we want this to equal $\alpha$ clearly we take $\lambda =k$.