Kullback-Leibler Divergence

Kullback-Leibler divergence measures how different one probability distribution is from another. It is often written as:

$$D_{KL}(P||Q)=\sum _{x}P(x)\log \frac{P(x)}{Q(x)}$$

For continuous variables:

$$D_{KL}(P||Q)=\int p(x)\log \frac{p(x)}{q(x)}dx$$

Intuition

KL divergence measures the extra information cost of using distribution $Q$ when the true distribution is $P$. If $Q$ assigns low probability to events that often happen under $P$, the KL divergence becomes large.

KL divergence is not symmetric:

$$D_{KL}(P||Q)\ne D_{KL}(Q||P)$$

This means it is not a distance metric in the strict mathematical sense.

Why The Direction Matters

The direction decides which distribution is treated as the reference. $D_{KL}(P||Q)$ punishes $Q$ heavily when it misses regions where $P$ has probability mass.

In Machine Learning, this matters because different objectives can produce different behavior. Some directions encourage covering all modes. Other directions encourage fitting one high-probability mode more tightly.

Relation To Cross Entropy

KL divergence is closely related to entropy and cross entropy:

$$H(P,Q)=H(P)+D_{KL}(P||Q)$$

If $P$ is fixed, minimizing cross entropy is equivalent to minimizing KL divergence. This is why classification losses often have an information-theoretic interpretation.

Diffusion Models

In diffusion models, KL divergence appears when comparing distributions in the forward noising process and the learned reverse denoising process. The model is trained so that its learned reverse transitions approximate the true reverse posterior.

The key idea is not that KL divergence is unique to diffusion. The key idea is that diffusion training needs a way to compare probability distributions over noisy latent variables.

Common Questions

• How does KL divergence work intuitively? It measures the penalty for using the wrong probability distribution.
• Why is it not symmetric? Because the expectation is taken under one distribution, so the two directions weight errors differently.
• How is marginal probability related? Marginal probability can define the observed-data distribution that a model tries to approximate, while KL divergence measures mismatch between distributions.