variational inference

A general approach to the problem of approximate inference in deep learning in Bayesian statistics It’s faster, though typically less accurate, than MCMC.

VI solves this via an optimization problem. We estimate $p(z|x)$ with

$$q^{\ast }(z)={\text{argmin}}_{q(z)\in \mathcal{Q}}{D}_{\text{KL}}(q(z)\Vert p(z|x)),$$

where $\mathcal{Q}$ is some tractable set of densities over $z$. Of course, this objective is also intractable since it depends on $p(x)$, which is what is difficult to compute in the first place. Instead we define

$$\ell (q):=\mathbb{E}\log p(z,x)-\mathbb{E}\log q(z),$$

and write

$$\begin{array}{rl}{D}_{\text{KL}}(q(z)\Vert p(z|x))& =\mathbb{E}[\log q(z)]-\mathbb{E}[\log p(z,x)]+\log p(x)\\ & =\log p(x)-\ell (q).\end{array}$$

So, minimizing the KL divergence is equivalent to maximizing $\ell$ up to a constant which doesn’t depend on $z$. The objective $\ell$ is called the “evidence lower bound”, or ELBO. This name comes from the fact that

$$\log p(x)=D_{\text{KL}}(q(z)\Vert p(z|x))+\ell (q)\ge \ell (q),$$

so $\ell (q)$ is a lower bound for the log-evidence.