Overview
Logistic Regression is a supervised learning algorithm used for classification tasks, despite having “regression” in its name. It predicts the probability that an instance belongs to a particular class.
Binary Logistic Regression
See Logistic Regression - Binary.
Multiclass Logistic Regression
See Logistic Regression - Multiclass.
Advantages
- Simple and interpretable: Coefficients show feature importance
- Probabilistic output: Provides confidence scores
- Fast training and prediction: Efficient for large datasets
- No hyperparameters (basic version)
- Works well with linearly separable data
- Less prone to overfitting than more complex models (with regularization)
Disadvantages
- Assumes linear decision boundary: Can’t capture complex patterns
- Sensitive to outliers: Extreme values can influence coefficients
- Requires feature engineering for non-linear relationships
- Multicollinearity issues: Correlated features can be problematic
- Not suitable for complex patterns (use tree-based or neural networks instead)
Regularization
See Logistic Regression - Regularization.
Assumptions
- Binary or ordinal outcome (for binary logistic regression)
- Linear relationship between log-odds and features
- Independence of errors (observations are independent)
- No multicollinearity among features
- Large sample size for stable estimates
Evaluation Metrics
- Accuracy: Overall correctness
- Precision: Of predicted positives, how many are correct?
- Recall: Of actual positives, how many did we find?
- F1-Score: Harmonic mean of precision and recall
- AUC-ROC: Area under ROC curve
- Average Precision: Area under Precision-Recall curve
Hyperparameters
| Parameter | Description | Common Values |
|---|---|---|
| C | Inverse of regularization strength | 0.01, 0.1, 1, 10, 100 |
| penalty | Regularization type | ’l1’, ‘l2’, ‘elasticnet’, ‘none’ |
| solver | Optimization algorithm | ’lbfgs’, ‘liblinear’, ‘saga’ |
| max_iter | Maximum iterations | 100, 1000, 10000 |
| class_weight | Weight for imbalanced classes | None, ‘balanced’, dict |
When to Use
Use Logistic Regression when:
- Need interpretable model (coefficients show feature importance)
- Decision boundary is roughly linear
- Need probability estimates
- Dataset is not too large
- Baseline model for comparison
- Text classification with bag-of-words
Avoid Logistic Regression when:
- Highly non-linear relationships
- Many interaction terms needed
- Very high-dimensional data (use regularized version)
- Complex patterns (use tree-based or neural networks)
Implementation Example
See Logistic Regression - Implementation Example.
Logistic Regression Vs Other Algorithms
| Algorithm | Decision Boundary | Speed | Interpretability |
|---|---|---|---|
| Logistic Regression | Linear | Fast | High |
| SVM | Linear/Non-linear | Medium | Medium |
| Decision Tree | Non-linear (axis-aligned) | Fast | High |
| Random Forest | Non-linear | Medium | Medium |
| Neural Network | Non-linear | Slow | Low |
| [[K Nearest Neigbour | KNN]] | Non-linear | Slow (prediction) |
Practical Considerations
- Scale features: Use StandardScaler or MinMaxScaler
- Handle imbalanced data: Use
class_weight='balanced'or resampling - Check linearity: Plot log-odds vs features
- Feature engineering: Create polynomial features if needed
- Regularization: Always use for high-dimensional data
- Cross-validation: Tune C parameter
- Interpret coefficients: Positive = increases probability, negative = decreases
References
- Multiclass Logistic Regression - GeeksforGeeks
- Scikit-learn documentation
- Introduction to Statistical Learning (ISLR)
- Pattern Recognition and Machine Learning (Bishop)