game-theoretic hypothesis testing

hypothesis-testingtesting-by-betting

Specific protocols of game-theoretic probability can be used to test hypotheses. This yields a fundamental tool in game-theoretic statistics.

Suppose we receive data $X_1,X_2,\cdots \sim P$ and we are testing the hypotheses:

$$H_0:P\in \mathbf{P}\text{ vs }{H}_1:P\in \mathbf{Q}.$$

This is a general framework which encapsulates both point nulls (eg ${\theta }^{\ast }=0$) and composite nulls (eg ${\theta }^{\ast }\in S$ for some set $S$). $\mathbf{P}$ and $\mathbf{Q}$ can be infinite sets of distributions, singletons, finite sets, etc.

The game proceeds as follows. Skeptic begins with wealth ${\mathcal{K}}_0=1$. For $t=1,\dots ,T$:

• Skeptic proposes a payoff function $S_t:\mathcal{X}\to [0,\infty ]$ such that

$${\mathbb{E}}_{X\sim P}[S_t(X)|{\mathcal{F}}_{t-1}]\le 1\forall P\in \mathbf{P}.$$

• Reality reveals an observation $X_t$.
• Skeptic updates her wealth as ${\mathcal{K}}_t={\mathcal{K}}_{t-1}{S}_t(X_t).$

As usual in game-theoretic statistics, this implies that ${\mathcal{K}}_t={\prod }_{i=1}^t{S}_i(X_i)$ is an e-process for $\mathbf{P}$. Under the null, the skeptic’s wealth is therefore not expected to grow. Indeed, by Ville’s inequality,

$$P(\forall t\ge 1:{\mathcal{K}}_t\ge 1/\alpha )\le \alpha \forall P\in \mathbf{P}.$$

Therefore, if we reject the null when the ${\mathcal{K}}_t$ exceeds $1/\alpha$, we have only an $\alpha$ probability of making an error, i.e., our type I error uniformly over time is bounded by $\alpha$. In other words, this is a level-$\alpha$ sequential test.

While this is framed as one-sample testing, we can also consider testing by betting—two-sample testing.

Payoff functions

Clearly, it is important to find good payoff functions. This constitutes a large area of study. While $S_t$ is designed to be a supermartingale under the null, we want it to grow as large as possible as quickly as possible under the alternative so that the null is rejected quickly.

Typically when designing betting strategies, we follow the approach of maximizing log-wealth, i.e., we want $\log ({\mathcal{K}}_t)$ to grow fast under the alternative. Though see growth rate conditions in sequential testing for a more general overview.

The payoff is admissible if $S_t^{\prime }\ge S_t$ a.s. under $P$ implies that $S_t^{\prime }=S_t$. If $S_t$ is admissible, then ${\mathbb{E}}_P[S_t(X)|{\mathcal{F}}_{t-1}]=1$ (as opposed to an inequality). This means the wealth process is a martingale (as opposed to a supermartingale). (This is easy to check: Set $S_t^{\prime }=S_t/{\mathbb{E}}_P[S_t|{\mathcal{F}}_{t-1}]$, which satisfies $\mathbb{E}[S_t^{\prime }|{\mathcal{F}}_{t-1}]=1$).

So how do we actually choose $S_t$? This choice depends on the problem at hand, whether it’s simple null vs simple alternative (testing by betting—simple vs simple), simple null vs composite alternative (testing by betting—simple vs composite), or composite null vs composite alternative (testing by betting—composite vs composite, in which case we use universal inference or the reverse information projection (RIPr).