Variational Inference

Variational inference (VI) approximates an intractable posterior $p(z|x)$ with a tractable distribution $q_{\varphi }(z)$ chosen from a parametric family $\mathcal{Q}$. It turns Bayesian inference into an optimization problem: find the $q$ closest to the true posterior under a divergence we can actually compute.

Reference reading: Variational Inference: A Review for Statisticians.

Setup

Conditional probability $p(z|x)$ is the probability that latent $z$ took a value given that observation $x$ was seen. As an everyday example: given that today is rainy, what is the probability that a person brings an umbrella? Here $z$ is bringing an umbrella and $x$ is a rainy day.

Bayes’ rule gives the posterior:

$$p(z|x)=\frac{p(x,z)}{p(x)}=\frac{p(x|z)p(z)}{\int p(x|z)p(z)dz}$$

The numerator is usually easy to evaluate. The denominator $p(x)$, called the marginal likelihood or evidence, requires integrating $z$ out of the joint and is intractable in any non-trivial latent-variable model.

Why Approximate

Sampling-based inference such as MCMC eventually converges, but it is slow for large datasets and high-dimensional latents. VI trades exactness for speed: it asks for the best member of a restricted family $\mathcal{Q}$ instead of the true posterior. This makes inference parametric, differentiable, and compatible with stochastic gradient training.

The ELBO

See Variational Inference - The ELBO.

Mean-Field Approximation

See Variational Inference - Mean-Field Approximation.

Reverse Vs Forward KL

VI minimizes $D_{KL}(q\Vert p)$, the reverse direction. This is mode-seeking: $q$ avoids regions where $p$ is small, so it tends to fit a single mode of a multimodal posterior tightly rather than averaging over modes. The forward direction $D_{KL}(p\Vert q)$ is mode-covering but requires samples from $p$, which is the thing we cannot do. The asymmetry is exactly the property discussed in Kullback-Leibler Divergence.

Amortized VI And VAEs

See Variational Inference - Amortized VI And VAEs.

Failure Modes

• Mean-field collapses correlations and tends to underestimate variance.
• A poorly expressive $\mathcal{Q}$ caps how well VI can approximate $p(z|x)$ regardless of optimization.
• Reverse-KL mode-seeking can hide other modes of a multimodal posterior; reported uncertainty looks confident even when the true posterior is broad.
• ELBO gradients are high-variance without the reparameterization trick or control variates.

One-line Interview Answer

Variational inference replaces an intractable posterior with the closest member of a parametric family by maximizing the ELBO, which equals the log-evidence minus the KL between the approximation and the true posterior.

Common Questions

• Why optimize the ELBO instead of the KL directly? The KL contains the intractable $\log p(z|x)$; the ELBO only needs the joint $p(x,z)$ and the chosen $q$.
• Why not just use MCMC? MCMC is asymptotically exact but slow at scale; VI is faster, parametric, and differentiable, at the cost of being approximate.
• What does “variational” mean? It refers to optimizing over a family of functions (here, distributions) rather than over scalars.