Johnny's Knowledge

K Nearest Neigbour - Preprocessing

Properties1
tagsmachine-learning, supervised-learning

Preprocessing Requirements

1. Feature Scaling (Critical!)

Why: Distance metrics are scale-dependent

from sklearn.preprocessing import StandardScaler
 
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Scaling methods:

  • StandardScaler: Mean=0, std=1 (default choice)
  • MinMaxScaler: Scale to [0, 1]
  • RobustScaler: Robust to outliers

2. Handling Missing Values

KNN cannot handle missing values directly:

  • Imputation (mean, median, mode)
  • KNN imputation (use KNN to predict missing values)

3. Dimensionality Reduction

For high-dimensional data:

  • PCA: Reduce dimensions while preserving variance
  • Feature selection: Keep only relevant features
  • t-SNE/UMAP: For visualization

Curse of Dimensionality

As dimensions increase:

  • All points become approximately equidistant
  • Nearest neighbors become less meaningful
  • Required data grows exponentially

Solutions:

  • Feature selection/engineering
  • Dimensionality reduction (PCA)
  • Use distance metrics suitable for high dimensions (cosine similarity)

Backlinks