Random Forest

Overview

Random Forest is an ensemble learning method that operates by constructing multiple Decision Tree during training and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

How It Works

1. Bootstrap Sampling (Bagging):

   • Create multiple random subsets of the training data by sampling with replacement
   • Each tree trains on a different bootstrap sample (~63% of original data)

2. Feature Randomness:

   • At each split, consider only a random subset of features
   • Typically √n features for classification, n/3 for regression (where n = total features)

3. Independent Tree Growth:

   • Each tree is grown to maximum depth without pruning
   • Trees are completely independent

4. Aggregation:

   • Classification: Majority vote across all trees
   • Regression: Average (mean) of all tree predictions

Key Concepts

See Random Forest - Key Concepts.

Advantages

• Reduced Overfitting: Averaging reduces variance compared to single tree
• Handles High-Dimensional Data: Works well with many features
• Robust to Outliers: Individual trees are less affected by noise
• No Feature Scaling Needed: Scale-invariant like decision trees
• Provides Feature Importance: Built-in feature ranking
• OOB Error Estimate: Free validation metric
• Handles Missing Values: Can maintain accuracy with missing data
• Parallel Training: Trees are independent, can train in parallel

Disadvantages

• Less Interpretable: Harder to explain than single decision tree
• Computationally Expensive: Training and prediction slower than single tree
• Memory Intensive: Stores multiple trees
• Still Biased for Imbalanced Data: May favor majority class
• Not Great for Extrapolation: Limited to range seen in training data

Hyperparameters

See Random Forest - Hyperparameters And Tuning.

Classification Vs Regression

See Random Forest - Classification Vs Regression.

Comparison with Other Methods

Method	Variance	Bias	Interpretability	Speed
Single Decision Tree	High	Low	High	Fast
Random Forest	Low	Low-Medium	Medium	Medium
Gradient Boosting	Low	Low	Medium	Slow
Linear Model	Low	High	High	Fast

Variants

See Random Forest - Variants.

Practical Considerations

1. Start Simple:

   • Use default parameters first
   • Check OOB error before cross-validation

2. Feature Engineering:

   • Random Forest benefits from feature engineering
   • Can handle collinearity well

3. Class Imbalance:

   • Use class_weight='balanced'
   • Or balanced sampling per tree

4. Interpretability:

   • Use feature importance plots
   • Consider SHAP values for better interpretation

5. Speed Optimization:

   • Use n_jobs=-1 for parallel processing
   • Reduce max_depth if too slow
   • Use fewer trees for rapid prototyping

When to Use

Use Random Forest when:

• Need a robust, accurate model with minimal tuning
• Have mixed feature types (categorical + numerical)
• Want feature importance rankings
• Interpretability is moderate priority
• Have enough computational resources

Avoid Random Forest when:

• Real-time predictions critical (slow)
• Memory is severely constrained
• Need perfect interpretability (use single tree)
• Linear relationships dominate (use linear models)
• Very high-dimensional sparse data (use linear models or neural nets)

Implementation Example

See Random Forest - Implementation Example.

References

• Breiman, L. (2001). “Random Forests”
• Scikit-learn documentation
• Introduction to Statistical Learning (ISLR)