e-values enable post-hoc hypothesis testing

In traditional hypothesis testing we fix a significance level $\alpha \in (0,1)$ and look for tests $\varphi$ which have type-I error bounded by $\alpha$:

$$\underset{P\in \mathcal{P}}{\sup }P(\varphi (X)=1)\le \alpha ,$$

where $\mathcal{P}$ is the null. To do post-hoc hypothesis testing, we want to be able to quantify over $\alpha$, so we rewrite this as risk:

$$\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P\left[\frac{\varphi (X)}{\alpha }\right]\le 1.$$

(We make a similar move when viewing hypothesis testing through the lens of decision-theory: Neyman-Pearson paradigm with losses). To allow $\alpha$ to be data-dependent, we add a supremum over $\alpha$ inside the expectation (it is implicitly outside the expectation in the previous display):

$$\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P\left[\underset{\alpha \in (0,1)}{\sup }\frac{\varphi (X)}{\alpha }\right]\le 1.$$

We call a rule $\varphi$ which obeys the above inequality a post-hoc valid hypothesis test.

Suppose the test is based on an e-value, i.e., for a given $\alpha$, $\varphi (x)=1(E(x)\ge \alpha )$ for some e-variable $E$ for $\mathcal{P}$. Then $\varphi (x)/\alpha \le E(x)$ by case analysis on the numerator, so

$$\underset{P}{\sup }{\mathbb{E}}_P\left[\underset{\alpha }{\sup }\frac{1(E(X)\ge \alpha )}{\alpha }\right]\le \underset{P}{\sup }{\mathbb{E}}_P[E(X)]\le 1,$$

meaning that tests based on thresholding e-values e-values enable post-hoc hypothesis testing.

The other direction is true as well: If one has a post-hoc valid test that is based on some threshold $T$, i.e., $\varphi (x)=1(T(x)\ge \alpha )$, then $T$ must be an e-value. You can see this by taking $\alpha =T$ in the definition of post-hoc validity.

This was first pointed out in False discovery rate control with e-values by Wang and Ramdas (2022) and in Beyond Neyman-Pearson by Grunwald (2022).