Layer & Batch Normalization - Layer Normalization

Layer Normalization

What It Does

LayerNorm normalizes all features within each sample.

Given a 2D input matrix:

• Rows = batch samples
• Columns = features

LayerNorm normalizes each row (sample) independently across all features.

How It Works

For each sample $i$ in the batch:

1. Compute mean and variance across features:

$${\mu }_i=\frac{1}{d}\sum _{j=1}^d{x}_{ij}$$ $${\sigma }_i^2=\frac{1}{d}\sum _{j=1}^d(x_{ij}-{\mu }_i{)}^2$$

2. Normalize:

$${\hat{x}}_{ij}=\frac{x_{ij}-{\mu }_i}{\sqrt{{\sigma }_i^2+ϵ}}$$

3. Scale and shift (learnable parameters):

$$y_{ij}={\gamma }_j{\hat{x}}_{ij}+{\beta }_j$$

Code Example

import torch.nn as nn
 
model = nn.Sequential(
    nn.Linear(64, 128),
    nn.LayerNorm(128), # Normalize across 128 features for each sample
    nn.ReLU(),
    nn.Linear(128, 10)
)

Advantages

• Works with any batch size (even batch size = 1)
• Same behavior in training and inference
• Better for RNNs and Transformers
• No dependency on batch statistics
• More stable for small batches

Disadvantages

• Less regularization effect than BatchNorm
• May not work as well for CNNs as BatchNorm