Self Attention - Multi-Head Attention

Multi-Head Attention

Splitting $Q$, $K$, $V$ into $h$ smaller heads lets the model attend to different relationships in parallel: one head can specialize in syntax, another in coreference, another in long-range topic links. The outputs are concatenated and projected back. Multi-head attention with small per-head $d_k$ is usually preferred over single-head attention with large $d_k$ because variety of attention patterns matters more than per-head capacity.