f-divergence

A broad class of distributional distances. For a convex function $f$ with $f(1)=0$ and two probability measures $\rho ,\nu$ defined on a measurable space $(\Omega ,\mathcal{F})$, set

$$D_f(\rho \Vert \nu )={\mathbb{E}}_{\nu }\left[f\left(\frac{\text{d}\rho }{\text{d}\nu }(X)\right)\right]={\int }_{\Omega }f\left(\frac{\text{d}\rho }{\text{d}\nu }(x)\right)\nu (\text{d}x).$$

With different choices of $f$ we can obtain

• total variation distance
• KL divergence
• Hellinger distance
• chi-squared divergence

f-divergences are mostly a distinct class of divergence than IPMs, intersecting only that the TV distance. Taking

$$f(u)=\frac{u^{\alpha }-1}{\alpha (\alpha -1)},$$

results in the alpha-divergence. There is a lots of literature on fixed-time plug-in estimators for $f$-divergences. In the sequential setting, because $f$-divergences are convex by construction they can estimated using reverse submartingales; see confidence sequences for convex functionals.

$f$-divergences admit the following variational inequality: For all measurable $h:\Theta \to \mathbb{R}$ and all distributions $\rho ,\nu$ over $\Theta$,

$$D_f(\rho \Vert \nu )\ge {\mathbb{E}}_{\rho }h(\theta )-{\mathbb{E}}_{\nu }{f}^{\ast }(h(\theta )),$$

where $f^{\ast }$ is the convex conjugate of $f$: $f^{\ast }(y)={\sup }_{x\in \mathbb{R}}\{xy-f(x)\}$. Equality holds for any $h$ in the subdifferential $\partial f(\text{d}\rho /\text{d}\nu )$. For the KL divergence, the Donsker-Varadhan variational formula strengthens this variational inequality.