Overview
Normalization techniques stabilize and accelerate training by normalizing inputs to each layer. The two main types are Batch Normalization and Layer Normalization, which differ in what they normalize.
Quick Comparison
| Aspect | Batch Normalization | Layer Normalization |
|---|---|---|
| Normalizes across | Batch dimension (samples) | Feature dimension (within sample) |
| Target | Each feature across batch | All features within one sample |
| Dependency | Requires batch statistics | Independent of batch size |
| Best for | CNNs, large batches | RNNs, Transformers, small batches |
| Training vs Inference | Different behavior | Same behavior |
Batch Normalization
See Layer & Batch Normalization - Batch Normalization.
Layer Normalization
See Layer & Batch Normalization - Layer Normalization.
When to Use Which?
Use Batch Normalization When
- Working with CNNs (Convolutional Neural Networks)
- Have large batch sizes (>32)
- Training feedforward networks
- Want additional regularization effect
- Batch statistics are meaningful (e.g., images are IID)
Use Layer Normalization When
- Working with Transformers or RNNs
- Have small batch sizes or online learning (batch size = 1)
- Sequence lengths vary (RNNs, NLP tasks)
- Batch statistics are not meaningful
- Need consistent behavior between training and inference
Example Use Cases
| Task | Recommended |
|---|---|
| Image Classification (CNN) | Batch Norm |
| Object Detection | Batch Norm |
| Language Models (Transformer) | Layer Norm |
| Machine Translation (RNN) | Layer Norm |
| Time Series (varying batch) | Layer Norm |
| Few-shot Learning | Layer Norm |
Other Normalization Techniques
Instance Normalization
Normalizes each sample and each channel independently. Used in style transfer.
nn.InstanceNorm2d(num_features)Group Normalization
Divides channels into groups and normalizes within each group. Middle ground between LayerNorm and InstanceNorm.
nn.GroupNorm(num_groups, num_channels)Summary
Key Insight: The main difference is the normalization axis:
- BatchNorm: Normalizes across the batch dimension (each feature separately)
- LayerNorm: Normalizes across the feature dimension (each sample separately)
Choose based on:
- Architecture (CNN → BatchNorm, Transformer → LayerNorm)
- Batch size (Large → BatchNorm, Small → LayerNorm)
- Task requirements (Sequence models → LayerNorm)