Gaussian Mixture Model

A Gaussian Mixture Model, or GMM, is a probabilistic Clustering model that represents data as a weighted mixture of multiple Gaussian distributions.

Intuition

A single Gaussian is good for one mound-shaped distribution. If the data has multiple modes, a mixture of Gaussians can model each mode with a separate component. The mixture weight ${\pi }_k$ says how much component $k$ contributes to the full distribution.

Model

The probability density is:

$$p(x)=\sum _{k=1}^K{\pi }_k\mathcal{N}(x\mid {\mu }_k,{\Sigma }_k)$$

where:

• ${\pi }_k$ is the mixture weight for component $k$.
• ${\mu }_k$ is the component mean.
• ${\Sigma }_k$ is the component covariance.
• The component identity is a latent variable because it is not directly observed.

Training

GMM parameters are usually estimated with Maximum Likelihood Estimation. Directly maximizing the likelihood is difficult because the component assignment is hidden, so GMM commonly uses the Em Algorithm:

1. E-step: estimate the probability that each point belongs to each Gaussian component.
2. M-step: update mixture weights, means, and covariances using those probabilities.

GMM Vs K-means

GMM is often described as the soft version of K-Mean Clustering. K-means assigns each point to one cluster. GMM gives each point a probability of belonging to each component, which is useful for overlapping clusters or uncertain boundaries.

Failure Modes

• The number of components $K$ must be chosen.
• Training can be sensitive to initialization.
• Components can collapse or overfit when covariance estimates are poorly regularized.
• A Gaussian assumption may fit poorly when clusters are non-elliptical.

One-line Interview Answer

A GMM models data as a weighted mixture of Gaussian components and uses EM to learn soft probabilistic cluster assignments.

Reference

• GMM derivation for diagonal covariance matrices