Johnny's Knowledge

Data Cleaning

Properties1
tagsdata-science, python

Data Cleaning Overview

Data cleaning is the process of preparing raw data for analysis by handling missing values, outliers, duplicates, and inconsistencies.

Best Practice: Always create a copy of the original dataset before cleaning.

# Create a copy to preserve original data
df_clean = df.copy()

Why? If you don’t, you lose the ability to check the original data or recover from mistakes.

Rename Columns

df.rename(columns={'oldName1': 'newName1', 'oldName2': 'newName2'})

Missing Value Handling

See Data Cleaning - Missing Value Handling.

Outlier Detection

Quick Methods

# Using IQR method
Q1 = df['column'].quantile(0.25)
Q3 = df['column'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
 
# Filter outliers
df_no_outliers = df[(df['column'] >= lower_bound) & (df['column'] <= upper_bound)]

Check Duplicates

# Check for duplicates
df.duplicated().sum()
 
# View duplicate rows
df[df.duplicated()]
 
# Remove duplicates
df.drop_duplicates(inplace=True)
 
# Remove duplicates based on specific columns
df.drop_duplicates(subset=['column1', 'column2'], inplace=True)

Data Type Conversion

# Convert to numeric (errors='coerce' converts invalid values to NaN)
df['column'] = pd.to_numeric(df['column'], errors='coerce')
 
# Convert to datetime
df['date'] = pd.to_datetime(df['date'])
 
# Convert to category (saves memory)
df['category_col'] = df['category_col'].astype('category')

Date Operations

Extract Date Components

# Extract year, month, day
df['year'] = df['Date Attribute'].dt.year
df['month'] = df['Date Attribute'].dt.month
df['day'] = df['Date Attribute'].dt.day
df['dayofweek'] = df['Date Attribute'].dt.dayofweek  # Monday=0, Sunday=6
 
# Extract time components
df['hour'] = df['DateTime'].dt.hour
df['minute'] = df['DateTime'].dt.minute

Date Calculations

# Calculate age from birthdate
from datetime import datetime
df['age'] = (datetime.now() - df['birthdate']).dt.days // 365
 
# Calculate time difference
df['days_diff'] = (df['end_date'] - df['start_date']).dt.days

Column Reordering

Move Target to Last Column

# Move target to the last column
cols = train.columns.tolist()
# Remove the 'target' column from the list
cols.remove('target')
# Append the 'target' column to the end of the list
cols.append('target')
# Reindex the DataFrame with the new column order
train = train[cols]

Reorder Specific Columns

# Put specific columns first
cols = ['id', 'name', 'age'] + [col for col in df.columns if col not in ['id', 'name', 'age']]
df = df[cols]

The inplace Parameter

inplace=True

  • Pandas doesn’t create a new DataFrame
  • Performs the operation directly on the original DataFrame
  • Returns None
  • Saves memory but can’t undo

inplace=False (default)

  • Pandas operations return a new copy of the DataFrame with changes applied
  • The original DataFrame remains untouched
  • Can chain operations
# With inplace=True (modifies original)
df.dropna(inplace=True)
 
# Without inplace (creates new DataFrame)
df_clean = df.dropna()

Data Validation

# Check data types
df.dtypes
 
# Check for negative values (if shouldn't exist)
df[df['price'] < 0]
 
# Check value ranges
df['age'].describe()
 
# Check unique values (for categorical)
df['category'].value_counts()

Export Cleaned Data

# Save to CSV
df.to_csv('cleaned_data.csv', index=False)
 
# For test dataset predictions (first row is id 0)
predictions_df = pd.DataFrame({'id': range(len(predictions)),
                                'prediction': predictions})
predictions_df.to_csv('predictions.csv', index=False)

Backlinks