Key Concepts
Bootstrap Aggregating (Bagging)
- Reduces variance by averaging predictions
- Each tree sees slightly different data
- Out-of-Bag (OOB) samples (~37%) can be used for validation
Out-of-Bag (OOB) Error
- Each tree is tested on samples it didn’t see during training
- Provides a built-in cross-validation estimate
- No need for separate validation set
Feature Importance
Random Forest can measure feature importance:
- Mean Decrease Impurity (MDI): Average decrease in node impurity across all trees
- Permutation Importance: Decrease in accuracy when feature values are shuffled