Self Attention

Self-attention is a mechanism that lets every position in a sequence attend to every other position by learning three projections of the input — Query, Key, Value — and aggregating Values weighted by Query-Key similarity.

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$$

Intuition

Each token plays three roles at once. Its Query is what it is looking for, its Key is what it offers to others, and its Value is the information it would contribute if matched. Attention is just matching queries to keys, then aggregating values by the match strength.

Compared to RNNs, every position can reach every other position in one step rather than through a chain of hidden states. Compared to convolutions, the receptive field is the full sequence by default rather than a fixed window.

How It Works

1. Project the input $X$ into three matrices: $Q=X{W}_Q$, $K=X{W}_K$, $V=X{W}_V$.
2. Score every query against every key with a dot product: $S=Q{K}^{\top }$.
3. Scale by $\sqrt{d_k}$ to keep the variance of $S$ stable as $d_k$ grows.
4. Apply softmax row-wise so each query’s scores form a probability distribution over keys.
5. Take a weighted sum of values: output is softmax weights times $V$.

Why The $\sqrt{d_k}$ Scale

Dot products grow with dimension, causing large magnitudes that push softmax into saturation (near one-hot), which leads to Vanishing Gradient. Scaling stabilizes the variance of the pre-softmax scores so gradients stay informative. The same variance-stabilization concern motivates layer normalization inside transformer blocks.

Multi-Head Attention

See Self Attention - Multi-Head Attention.

Failure Modes

• Without scaling, softmax saturates for large $d_k$ and gradients collapse.
• Without softmax, weights are unbounded and possibly negative, breaking the probabilistic-aggregation interpretation and destabilizing training.
• Cosine similarity instead of dot product loses magnitude information that learned projections rely on; scaled dot product is the standard compromise.
• Quadratic cost in sequence length: attention matrix is $n\times n$, so long sequences need approximations (sparse, linear, or chunked attention).

One-line Interview Answer

Self-attention computes an output for each token as a softmax-weighted sum of all tokens’ Values, with the weights coming from scaled dot products between that token’s Query and every other token’s Key.

Common Questions

• Why scale by $\sqrt{d_k}$? To prevent dot-product variance from saturating softmax and causing Vanishing Gradient.
• What if we remove softmax? Weights become unbounded and negative; attention loses probabilistic aggregation and becomes unstable.
• Why dot product instead of cosine similarity? Dot product is a single matmul and combines naturally with learned $W_Q,W_K$ projections; scaling compensates for magnitude sensitivity.
• What are Q, K, V intuitively? Query is what a token is looking for, Key is what it offers, Value is what it contributes when matched.