alpha-divergence

There are three kinds of $\alpha$-divergences: the generic version, the Renyi $\alpha$-divergence and the Tsallis $\alpha$-divergence.

If $\text{d}\rho$ and $\text{d}\nu$ are the densities of $\rho$ and $\nu$ respectively, the generic $\alpha$-divergence is defined as

$$D_{\alpha }(\rho \Vert \nu )=\frac{1}{\alpha (\alpha -1)}\left(\int \text{d}{\rho }^{\alpha }(x)\text{d}{\nu }^{1-\alpha }(x)\text{d}x-1\right),$$

for any $\alpha \ne 0,1$. This is a special case of an f-divergence, thus admits the same variational inequality. This also means it’s a convex distributional distance. It convergences to the KL divergence as $\alpha \to 1$, the reverse KL as $\alpha \to 0$ and to the Hellinger distance at $\alpha =0.5$. For $\alpha =2$ it’s the chi-squared divergence.

Renyi $\alpha$-divergence

The Renyi $\alpha$-divergence is

$$R_{\alpha }(\rho \Vert \nu )=\frac{1}{\alpha -1}\log \int \text{d}{\rho }^{\alpha }(x)\text{d}{\nu }^{1-\alpha }(x)\text{d}x.$$

This obeys: $\alpha <0\Rightarrow R_{\alpha }\le 0$, $\alpha =0\Rightarrow R_{\alpha }=0$, $\alpha >0\Rightarrow R_{\alpha }\ge 0$, and $\alpha =1$ this converges to the KL divergence. The Renyi divergence admits the variational inequality

$$D_{\alpha }(\rho \Vert \nu )\ge \frac{\alpha }{\alpha -1}\log {\mathbb{E}}_{\rho }h(\theta )-\log {\mathbb{E}}_{\nu }h(\theta {)}^{\frac{\alpha }{\alpha -1}},$$

for all measurable $h$ and $\alpha >0$, $\alpha \ne 1$.

Tsallis $\alpha$-divergence

The Tsallis divergence is

$$T_{\alpha }(\rho )=\frac{1}{\alpha -1}\left(\int \text{d}{\rho }^{\alpha }(x)\text{d}{\nu }^{1-\alpha }(x)\text{d}x-1\right).$$

Like the Renyi-divergence, this obeys $\alpha <0\Rightarrow T_{\alpha }\le 0$, $\alpha =0\Rightarrow T_{\alpha }=0$, $\alpha >0\Rightarrow T_{\alpha }\ge 0$.