KL divergence

For probability measures $\rho ,\nu$ on a measurable space $(\Omega ,\mathcal{F})$, the KL-divergence is

$$D_{\text{KL}}(\rho \Vert \nu )={\mathbb{E}}_{\rho }\left[\log \left(\frac{\text{d}\rho }{\text{d}\nu }(X)\right)\right]=\int \log \left(\frac{\text{d}\rho }{\text{d}\nu }(x)\right)\rho (\text{d}x),$$

if $\rho \ll \nu$ and $\infty$ otherwise. The KL-divergence is an f-divergence resulting from $f(x)=x\log x$.

Donsker-Varadhan variational formula

Let $h:\Theta \to \mathbb{R}$ be measurable and $\nu$ be a distribution over $\Theta$, then

$$\log {\mathbb{E}}_{\nu }\exp h(\theta )=\underset{\rho }{\sup }\left\{{\mathbb{E}}_{\rho }h(\theta )-D_{\text{KL}}(\rho \Vert \nu )\right\}.$$

This strengthens the general variational inequality for f-divergences. This is a crucial ingredient in PAC-Bayes bounds, which are typically stated in terms of the KL-divergence.