confidence sequences

A confidence sequence can be thought of as a sequence of sequentially valid confidence intervals). Formally, $(C_t{)}_{t\ge 1}$ is a $(1-\alpha )$-CS for a parameter $\theta$ if

$$\mathbb{P}(\exists t\ge 1:\theta \notin C_t)\le \alpha .$$

That is, a CS is a time-uniform confidence interval. It avoids the known problem with confidence intervals that, if they are computed at different times, they risk contradicting themselves (eg an interval computed at time $t_1$ might not overlap with one at time $t_2$. )

Like confidence intervals, confidence sequences have frequentist guarantees (frequentist statistics), not Bayesian ones. Whether there exist anytime-valid credible intervals (the Bayesian notion of confidence interval) is an interesting question.

Equivalent definitions are:

• $\mathbb{P}({\theta }^{\ast }\in C_{\tau })\ge 1-\alpha$ for any stopping time $\tau$
• $\mathbb{P}({\theta }^{\ast }\in C_T)\ge 1-\alpha$ for all random times $T$. A random time could be a function of a stopping time, for instance (eg $\tau /2$).

These are not obviously equivalent definitions. They have to be proved. For instance, the same properties do not hold with expected values. See lemmas 2 and 3 here and the anytime-valid notes.

Confidence sequences are common tools in safe, anytime-valid inference (SAVI).

Many confidence sequences are construct from applying Ville’s inequality to supermartingales or test-martingales]. It is known how to construct non-trivial confidence sequences for a variety of problems. A few include:

• bounded, non-parametric mean estimation in $\mathbb{R}$ (estimating means by betting) and in ${\mathbb{R}}^d$ (CSs via universal gambling strategies), and in Banach spaces (via empirical Bernstein bounds).
• Mean estimation under light-tailed sub-psi conditions: See Howard et al. 2021.
• Mean estimation for heavy-tailed scalar observations (see discussion in Catoni-Giulini M-estimator) with possibly infinite variance.
• Estimating Gaussian means with unknown variance
• Light and heavy-tailed (finite variance) mean estimation in ${\mathbb{R}}^d$, using the variational approach to concentration.
• confidence sequences for quantiles

There is also an extension of confidence sequences to the asymptotic regime: asymptotic confidence sequences.

Refs

• Time-uniform, nonparametric, nonasymptotic confidence sequences by Howard et al.