Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation is a method for estimating the parameters of a statistical model. It finds the parameter values that maximize the likelihood function, making the observed data most probable.

Concept

Given a dataset and a parametric model, MLE finds the parameter values θ that maximize:

$$L(\theta |x)=P(x|\theta )$$

or equivalently, maximize the log-likelihood:

$$\log L(\theta |x)=\sum _{i=1}^n\log P(x_i|\theta )$$

Steps

1. Define the likelihood function: Based on the assumed probability distribution
2. Take the logarithm: Convert to log-likelihood for easier computation
3. Differentiate: Take the derivative with respect to parameters
4. Set to zero: Solve for critical points
5. Verify maximum: Check second derivative or use other methods

Common Examples

Normal Distribution

For data from N(μ, σ²):

• MLE of μ = sample mean
• MLE of σ² = sample variance (biased estimator)

Bernoulli Distribution

For binary data:

• MLE of p = number of successes / total trials

Properties

• Consistency: Converges to true value as sample size increases
• Asymptotic normality: Distribution approaches normal for large samples
• Efficiency: Achieves lowest variance among unbiased estimators (asymptotically)
• Invariance: MLE of g(θ) is g(MLE of θ)

Applications

• Parameter estimation in statistical models
• Training Machine Learning models (notably GMM via the Em Algorithm)
• Fitting probability distributions