distributional distance

Functions which capture “distances” between probability distributions. They may not be metrics in the formal sense (metric space) (eg perhaps they’re not symmetric as in the KL divergence).

Examples:

• KL divergence
• Wasserstein distance
• KS distance
• total variation distance
• Hellinger distance
• chi-squared divergence
• f-divergence (encapsulates several of the above)
• Fisher information distance

Some definitions

Different authors have different notation: Some write distances as functions of the distributions themselves, eg $\rho (P_1,P_2)$, and some write them as functions of random variables, $\rho (X_1,X_2)$.

A metric $\rho$ is regular if $\rho (X+Z,Y+Z)\le \rho (X,Y)$ for any $Z$ independent of $X$ and $Y$. This captures the notion that blurring observations by independent noise makes them harder to distinguish, i.e., decreases the distance between them.

Regularity is equivalent to sub-additivity:

$$\rho (X_1+X_2,Y_1+Y_2)\le \rho (X_1,Y_1)+\rho (X_2,Y_2),\text{for }{X}_1\perp X_2,Y_1\perp Y_2.$$

A metric is homogeneous of order $s$ if

$$\rho (cX,cY)=|c{|}^s\rho (X,Y)\text{for all }c\in \mathbb{R}.$$

Ideal metrics of order $s$ are simultaneously regular and homogeneous of order $s$. These come up in the study of central limit theorems (see quantitative CLT template with ideal metrics).