truncation-based estimators

In 2018, Catoni and Giulini proposed a simple estimator in ${\mathbb{R}}^d$: Simply truncate the observations and take the empirical mean of the result. In particular, they consider

$$\hat{\mu }=\frac{1}{n}\sum _{i\le n}\alpha (X_i)X_i,$$

where

$$\alpha (X):=\frac{\min \{\lambda \Vert X\Vert ,1\}}{\lambda \Vert X\Vert }=\min \{1,(\lambda \Vert X\Vert {)}^{-1}\},$$

for some $\lambda >0$. Estimators of this type are also called thresholding estimators. The Catoni-Giulini estimator.

This has a rate of $O(\sqrt{v^2\log (1/\delta )/n})$ where $\mathbb{E}{\Vert X-\mu \Vert }^2\le v^2$. The rate is therefore not quite sub-Gaussian, but close. See the discussion by Lugosi and Mendelson (Section 3.3) for a clear exposition about the precise rates of the Catoni-Giulini estimator. We gave a sequential version of this estimator. The guarantees for the original Catoni-Giulini estimator rely on proofs using PAC-Bayes. Blog post on this estimator is here. See variational approach to concentration for a generalization of this proof technique to other estimators.

As presented, the Catoni-Giulini truncation estimator requires a bound on the raw second moment $\mathbb{E}{\Vert X\Vert }^2$. This is less desirable than a central moment assumption, i.e., a bound on $\mathbb{E}{\Vert X-\mu \Vert }^2$, since the latter is immune to translations of the data. As noted by Lugosi and Mendelson, this translation invariance can be overcome by sample splitting: use some logarithmic number of points to construct a naive estimate $m$ of the mean, and then center the Catoni-Giulini estimator around $m$.

This estimator was extended to Banach spaces (see also concentration in Banach spaces) by Whitehouse et al. They (we) also show that the estimator handles martingale dependence and is time-uniform, and also can handle a central moment assumption (instead of the raw moment), and also handles infinite variance (requires only a central $p$-th moment for $p>1$). Interestingly, they don’t use a PAC-Bayes analysis, but instead employ the Pinelis approach to concentration.