What You Will Learn
INTERMEDIATE
- Self-attention in one sentence (then unpacked properly)
- Q, K, V matrices without the math terror
- Multi-head attention: why "more heads = better"
Self-Attention in One Sentence
Self-attention is a mechanism where each word in a sentence decides how much to "pay attention" to every other word, so that the meaning of each word is enriched by the context around it. That's it. Everything else — Q/K/V matrices, softmax scaling, multi-head — is just an efficient way to compute this attention.
Let's make this concrete. Take the sentence: "The bank of the river was muddy." The word bank is ambiguous — it could mean a financial institution or the side of a river. How does the model know which meaning to use? Because bank "attends to" river and muddy, which together push its meaning toward the river-side sense. Without attention, the model would have to guess based on the word alone. With attention, the meaning of every word is computed dynamically from the surrounding words.
The Q, K, V Story (Without the Math Terror)
Now the technical version. Each token in your input has three things computed from it: a Query (Q), a Key (K), and a Value (V). All three are vectors — lists of numbers — produced by multiplying the token's embedding by three learned weight matrices (also called Q, K, V).
Here's the intuition that makes it click:
- Query (Q): "What am I looking for?" — When the word
bankwants to figure out its meaning, its Query is something like "I'm ambiguous; I need context about bodies of water or money." - Key (K): "What do I have?" — Each other word has a Key that advertises what it offers.
river's Key might say "I'm about water, nature, geography." - Value (V): "Here's what I'll give you." — When
bankandriverhave a high Q-K match,riversends its Value (its semantic content) to enrichbank's representation.
The match between a Query and a Key is just a dot product (cosine similarity, basically). High dot product = strong attention = lots of value flowing from that word.
The Math (Just Enough, I Promise)
The full attention formula is shorter than you think:
# Self-attention in 4 lines of NumPy
import numpy as np
def attention(Q, K, V):
# Q, K, V are matrices of shape (seq_len, d_k)
scores = Q @ K.T / np.sqrt(K.shape[-1]) # how much each Q matches each K
weights = softmax(scores, axis=-1) # normalize to probabilities
return weights @ V # weighted sum of values
The 1/sqrt(d_k) scaling is just to keep the dot products from getting too large (which would make softmax saturate and gradients vanish). Softmax turns the raw scores into a probability distribution (sums to 1). The final multiply gives you a weighted blend of all the Values — this is the new, context-enriched representation of the current word.
Multi-Head Attention: Why More Heads = Better
A single attention mechanism can only learn one type of relationship at a time. But language has many: subject-verb agreement, adjective-noun modification, negation scope, coreference (which "he" or "she" refers to), long-range dependencies, etc. The Transformer solves this with multi-head attention: instead of one Q/K/V, you have N (typically 8, 16, or 96) parallel "heads", each with its own learned Q/K/V matrices.
Each head learns a different relationship. Head 1 might focus on subject-verb agreement. Head 2 on adjective-noun. Head 3 on long-range dependencies. And so on. The outputs of all heads are concatenated and passed through a final linear layer to mix them. This is why models with more heads generally do better — they can model more relationships simultaneously.
| Model | # Heads | Head Dim | Total Attn Dim |
|---|---|---|---|
| GPT-2 small | 12 | 64 | 768 |
| GPT-3 (175B) | 96 | 128 | 12,288 |
| Llama 3 8B | 32 | 128 | 4,096 |
| Llama 3 70B | 64 | 128 | 8,192 |
| GPT-4 (est.) | 120 | 128 | 15,360 |
Why Attention Is Fast (and Slow)
Self-attention is O(n²) in sequence length — every word attends to every other word, so a 100K-token context means 10 billion attention scores to compute. This is why long-context models (1M+ tokens) require clever tricks like sparse attention, sliding window attention, or context compression.
The flip side: attention is fully parallelizable across positions (unlike RNNs which must process sequentially). This is why Transformers train much faster on GPUs than RNNs did — every token's attention can be computed in parallel.
Practical Exercise
🎯 Key Takeaways
- Self-attention lets every word look at every other word to gather context.
- Q (query), K (key), V (value) — like searching a library: query says what you want, key advertises what's there, value is the content.
- Multi-head attention learns multiple relationships in parallel — more heads = richer understanding.
- Attention is O(n²) in sequence length — this is why long context is expensive.
- KV caching makes generation fast by not recomputing old K and V vectors.
Comments
Comments
Post a Comment