Distance Metrics And Similarity Measures

Overview

Distance metrics and similarity measures are fundamental concepts in machine learning, used extensively in:

• Clustering algorithm K-means, DBSCAN, Hierarchical)
• K-Nearest Neighbors (KNN)
• Anomaly Detection
• Information retrieval
• Recommendation systems

Distance Metrics

Euclidean Distance

The most common distance metric, measuring the straight-line distance between two points in Euclidean space.

Formula (2D):

$$d(p,q)=\sqrt{(p_1-q_1{)}^2+(p_2-q_2{)}^2}$$

General Formula (n-dimensional):

$$d(p,q)=\sqrt{\sum _{i=1}^n(p_i-q_i{)}^2}$$

When to use:

• When the magnitude of differences matters
• For continuous numerical data
• Default choice for many ML algorithms
• Works well when features are on similar scales

Properties:

• Sensitive to scale (requires normalization)
• Affected by outliers
• Assumes isotropic (circular/spherical) distributions

Manhattan Distance

Also known as Taxicab distance or L1 distance. Measures the sum of absolute differences between coordinates.

Formula:

$$d(p,q)=\sum _{i=1}^n|p_i-q_i|$$

When to use:

• When movement is restricted to grid-like paths
• For high-dimensional data
• Less sensitive to outliers than Euclidean
• When you want a more robust metric

Example:
Think of navigating city blocks - you can’t go diagonally through buildings, you must follow the streets.

Reference: What is Manhattan Distance? A Deep Dive | DataCamp

Minkowski Distance

A generalization of both Euclidean and Manhattan distances.

Formula:

$$d(p,q)={\left(\sum _{i=1}^n|p_i-q_i{|}^p\right)}^{1/p}$$

Where:

• p = 1: Manhattan distance
• p = 2: Euclidean distance
• p → ∞: Chebyshev distance (maximum coordinate difference)

When to use:

• When you want flexibility in choosing distance behavior
• Useful for experimenting with different distance metrics
• Can be tuned as a hyperparameter

Similarity Measures

See Distance Metrics And Similarity Measures - Similarity Measures.

Choosing the Right Metric

Metric	Best For	Pros	Cons
Euclidean	Continuous data, low dimensions	Intuitive, commonly used	Sensitive to scale and outliers
Manhattan	Grid-like data, high dimensions	Robust to outliers	Less intuitive
Minkowski	Experimental tuning	Flexible, generalizable	Requires parameter tuning
Cosine	Text, sparse high-dim data	Magnitude-independent	Ignores vector length
Jaccard	Sets, binary features	Simple for categorical data	Only for set-like data

Practical Applications

K-Means Clustering

Typically uses Euclidean Distance, but can use others like Manhattan Distance depending on the data and task.

DBSCAN

Uses distance metrics (typically Euclidean) with the $ϵ$ parameter to define neighborhoods.

Hierarchical Clustering

Calculates distances (Euclidean Distance) and uses different linkage methods (Single, Complete, Average).

K-Nearest Neighbors (KNN)

Relies heavily on distance metrics to find the k closest neighbors.

Implementation Tips

1. Normalize/Standardize features when using Euclidean or Minkowski distance
2. Consider sparsity - Cosine similarity works well for sparse data
3. Think about semantics - Does magnitude matter in your application?
4. Experiment - Try multiple metrics and compare results
5. Computational cost - Manhattan is faster to compute than Euclidean

References

• What is Manhattan Distance? A Deep Dive | DataCamp
• Cosine similarity vs dot product - Data Science Stack Exchange
• Vector Similarity Explained | Pinecone