Variational Inference - The ELBO

The ELBO

VI minimizes the KL divergence from the approximation to the true posterior:

$$q^{\ast }(z)=\arg \underset{q\in \mathcal{Q}}{\min }{D}_{KL}\left(q(z)\Vert p(z|x)\right)$$

The KL is itself intractable because it contains $\log p(z|x)$, which is what we wanted to compute. The standard trick rewrites it:

$$\log p(x)={\mathbb{E}}_{q(z)}\left[\log \frac{p(x,z)}{q(z)}\right]+D_{KL}\left(q(z)\Vert p(z|x)\right)$$

The first term is the Evidence Lower Bound (ELBO):

$$\mathcal{L}(q)={\mathbb{E}}_{q(z)}\left[\log p(x,z)-\log q(z)\right]$$

Since $\log p(x)$ is a constant w.r.t. $q$, minimizing the KL is equivalent to maximizing the ELBO. Both quantities only require the joint $p(x,z)$ and the chosen $q$, both of which are tractable.