Johnny's Knowledge

Anomaly Detection - Project Implementation

Properties1
tagsmachine-learning, anomaly-detection

Project Implementation

Pipeline

Data Preprocessing

Handling Missing Values

Handle missing values for time-series data with forward fill and backward fill (see Data Cleaning)

StandardScaler(with_mean=False)

StandardScaler(with_mean=False) only scales the data by the standard deviation and does not center it by subtracting the mean.

Without:

scaled_value = (original_value - mean) / standard_deviation

With with_mean=False:

scaled_value = original_value / standard_deviation

The resulting data has a standard deviation of 1, but its mean will not be 0.

Why use it? To preserve sparsity in the data.

Sparse data contains a lot of zeros.

The problem: If you take a sparse matrix full of zeros and subtract the mean from every element, almost all of those zeros will become non-zero values, which breaks the sparsity and data becomes a dense format. A dense matrix can consume more memory and slow down computations significantly.

By scaling data without subtracting the mean, we preserve zeros in sparse data.

OneHotEncoder(sparse_output=False)

When one-hot encoding a categorical feature, the resulting matrix is mostly filled with zeros. There are two ways to store them:

1. Sparse Matrix (sparse_output=True, the Default)

This format is highly efficient because it only stores the locations of the non-zero values (the 1s). It assumes everything else is zero.

  • Analogy: Imagine a mostly empty movie theater. A sparse format is like having a list that only says, “Seat C7 is taken, Seat F12 is taken.” You don’t waste time listing every single empty seat.
  • Best for: Datasets with many categories (high cardinality). This can save a huge amount of memory and speed up calculations.

2. Dense Matrix (sparse_output=False)

This is a regular NumPy array. It explicitly stores every single value in memory, including all the zeros.

  • Analogy: A dense format is like having a full seating chart of the entire theater, with every seat explicitly marked as either “taken” or “empty.”
  • Best for: Datasets with very few categories, or when you need to easily view and manipulate the array for debugging or use with libraries that don’t support sparse matrices.

Will it affect model results? No, it’s just a matter of memory usage and computational speed.

ColumnTransformer

  • remainder="passthrough": Keep unselected columns for transformation unchanged
  • remainder="drop": Remove any column that is not selected for transformation
  • sparse_threshold: A value between 0 to 1 that decides if output is dense or sparse

Model Ensemble

  • Already experimented IForest with nearest neighbor (INNE) with minimal feature engineering
  • Takeaway: Change of model didn’t improve AP & AUC-ROC much compared with IForest with improved feature engineering
  • Action: Focus on feature engineering & preprocessing, then model after

Backlinks