Logistic Regression - Regularization

Regularization

Prevent overfitting by penalizing large coefficients.

L2 Regularization (Ridge)

$$J(w)=\text{Original Loss}+\lambda \sum _{j=1}^n{w}_j^2$$

• Shrinks coefficients toward zero
• Doesn’t set coefficients exactly to zero
• Good when all features are relevant

L1 Regularization (Lasso)

$$J(w)=\text{Original Loss}+\lambda \sum _{j=1}^n|w_j|$$

• Can set coefficients exactly to zero
• Performs feature selection
• Good for high-dimensional sparse data

Elastic Net

Combines L1 and L2:

$$J(w)=\text{Original Loss}+{\lambda }_1\sum _{j=1}^n|w_j|+{\lambda }_2\sum _{j=1}^n{w}_j^2$$