Missing Value Handling
Find Missing Values
# Count missing values per column
df.isnull().sum()
# Percentage of missing values
(df.isnull().sum() / len(df)) * 100
# Visualize missing data
import missingno as msno
msno.matrix(df)Imputation Methods
1. Remove Missing Data
# Drop rows with any missing values
df.dropna()
# Drop rows where specific column has missing values
df.dropna(subset=['column_name'])
# Drop columns with missing values
df.dropna(axis=1)2. Fill with Statistical Values
# Fill with mean
df['column'].fillna(df['column'].mean(), inplace=True)
# Fill with median (more robust to outliers)
df['column'].fillna(df['column'].median(), inplace=True)
# Fill with mode (for categorical data)
df['column'].fillna(df['column'].mode()[0], inplace=True)3. Forward Fill & Backward Fill (Time Series)
Usually used for time-series data missing value imputation.
Forward Fill (Ffill)
Forward fill uses the last known value forward to fill missing values. This is useful when you assume that the last observed value should remain until a new value is recorded.
df['var'].ffill()Backward Fill (Bfill)
Backward fill uses the next known value backward to fill missing values. This is useful when you assume that future observations should fill the earlier gaps.
df['var'].bfill()Note: They keep finding until they find a valid non-missing value to fill.
Example:
# Time series data with missing values
import pandas as pd
data = {'date': ['2024-01-01', '2024-01-02', '2024-01-03', '2024-01-04'],
'value': [10, None, None, 20]}
df = pd.DataFrame(data)
# Forward fill
df['ffill'] = df['value'].ffill() # [10, 10, 10, 20]
# Backward fill
df['bfill'] = df['value'].bfill() # [10, 20, 20, 20]