comparing forecasters by betting

Just as we can test forecasters, we can also compare two different forecasters who are making predictions over time. The setup follows that of game-theoretic hypothesis testing.

Let $S:\Delta (\mathcal{X})\times [0,1]\to \mathbb{R}$ be a proper scoring rule. Suppose forecaster 1 makes forecasts $p_t\in \Delta (\mathcal{X})$ at time $t$, and forecaster 2 makes forecasts $q_t\in \Delta (\mathcal{X})$ at time $t$. Define

$${\hat{\Delta }}_t=\frac{1}{t}\sum _{i\le t}(S(p_i,o_i)-S(q_i,o_i)),$$

which is the empirical difference between the forecasters performance, where $o_i$ is the outcome of the $i$-th forecast. We want to quantity the difference between ${\hat{\Delta }}_t$ and ${\Delta }_t$, the true expected difference between the forecasters:

$${\Delta }_t=\frac{1}{t}\sum _{i\le t}{\mathbb{E}}_{o_i\sim D_i}[S(p_i,o_i)-S(q_i,o_i)|{\mathcal{F}}_{i-1}],$$

where $({\mathcal{F}}_t)$ is the filtration capturing what’s happened so far. Depending on the behavior of $S$ (eg if it’s bounded, light-tailed etc) we can develop confidence sequences for the difference $|{\hat{\Delta }}_t-{\Delta }_t|$ or perform sequential hypothesis testing to determine if ${\Delta }_t\equiv 0$ (say). Depending on the behavior of $S$, we can form sub-psi process for $t({\Delta }_t-{\hat{\Delta }}_t)$, which lets apply much of the machinery of safe, anytime-valid inference (SAVI) to this problem.

Refs

• Comparing sequential forecasters, Choe and Ramdas.