Vanishing gradient is a training problem where gradients become extremely small (close to 0) during backpropagation, so earlier layers learn very slowly or stop learning entirely.
Why it Happens
During backpropagation, gradients are computed using the chain rule, which multiplies many derivatives:
If many of these derivatives are < 1, repeated multiplication makes the gradient shrink exponentially.
Intuition
- Think of passing a signal through many layers
- Each layer slightly “shrinks” the signal
- After many layers → signal becomes almost zero
Where it Commonly Occurs
- Deep neural networks (many layers)
- Recurrent neural networks (long sequences)
- Activation functions like sigmoid and tanh, whose derivatives ≤ 1 (often much smaller) shrink the signal at every layer
- Self Attention without proper scaling, where large dot products push softmax into a near one-hot distribution and its gradients collapse—a different mechanism (saturation) reaching the same outcome (negligible gradients)
Why it is a Problem
- Early layers stop updating
- Model fails to learn meaningful features
- Training becomes very slow or stuck
Simple Example
If each layer contributes a factor of 0.5:
After 10 layers, gradient is almost zero.
Solutions (High-level)
- ReLU activation (avoids saturation)
- Residual connections (used in Transformers)
- Proper initialization
- Normalization (LayerNorm, BatchNorm)
- LSTM/GRU for sequences
One-line Interview Answer
Vanishing gradient is when gradients shrink exponentially during backpropagation, preventing early layers from learning effectively.