Attention

Queries, keys, and values

Paper: Attention Is All You Need — Vaswani et al., 2017

Self-attention is the core mechanism that lets standard transformer LLMs exchange information across positions.

This section is the longest in the essay. Get it, and the rest of the transformer falls into place.

The setup

Imagine the model has read the prompt “the cat sat on the soft” and now has to predict what comes next. After the embedding step from the previous section, you have a sequence of 6 vectors: one per token. Call the embeddings $x_1, \ldots, x_6$, each of size $d_{\text{model}}$ (say 4096). Attention’s job is to produce a new sequence of 6 vectors $y_1, \ldots, y_6$ where each $y_i$ is informed by all the earlier $x_j$ it might find relevant. The one we care about most is the final position, $y_6$, for “soft”: after passing through every layer, its final hidden state will produce the logits used to sample the next token (with luck, “mat”).

(There’s one thing missing here that you might already be wondering about: nothing in this setup says where each token sits in the sequence. We’ll come back to that in section 6. For now, just imagine the model already knows.)

The first move is to project each input vector $x_i$ into three different vectors using three learned weight matrices: $W_Q$, $W_K$, and $W_V$.

Each query and key vector has dimension $d_k$; each value vector has dimension $d_v$. In the standard multi-head layout, these are usually equal and smaller than $d_{\text{model}}$. These are the famous queries, keys, and values.

The names are an information-retrieval analogy:

• The queryqueryA vector asking “what am I looking for in other tokens?”. Computed per token, used to score against keys.See in glossary → $q_i$ asks: “What information am I looking for?”
• The keykeyA vector saying “what I represent”. Compared against queries to compute attention scores.See in glossary → $k_j$ advertises: “Here’s what kind of information I contain.”
• The valuevalueA vector representing the content actually mixed into the output when a token gets attended to.See in glossary → $v_j$ holds: “Here’s the actual information I’ll contribute if someone picks me.”

Scoring: how much does $i$ care about $j$?

For every pair $(i, j)$, we compute an attention scoreattention scoreA single number sᵢⱼ measuring how much token i wants to attend to token j. Computed as the dot product of i's query vector and j's key vector (scaled by √d_k), then softmaxed across j so the weights for each i sum to 1. High score = i finds j relevant.See in glossary → by taking the dot productdot productA single number summarizing how aligned two vectors are. To compute a · b: multiply matching components and add the results. Positive means they point partly together, zero means they are at right angles, and negative means they point partly apart; what that means for the data depends on how the vectors were learned.See in glossary → of $q_i$ and $k_j$:

That is: multiply $q_i$ and $k_j$ component-by-component, then add up all the products into a single scalar. If the two vectors point in similar directions in their high-dimensional space, the matched components reinforce each other and the sum is large and positive. If they point in unrelated directions, positive and negative products cancel and the sum is near zero. A high score therefore means “token $i$‘s query lines up with token $j$‘s key”: token $i$ wants information from token $j$.

We then divide by $\sqrt{d_k}$ to keep the numbers from blowing up. This is the “scaled” in scaled dot-product, and it’s worth understanding where that exact factor comes from.

$d_k$ is the head dimension: the length of each query and key vector. In the standard multi-head layout, the model width $d_{\text{model}}$ is usually split across $h$ attention heads, so $d_k$ is typically $d_{\text{model}} / h$. With 512 dimensions and 8 heads, $d_k = 64$.

Why does the score need scaling at all? Look back at the sum in $s_{ij} = \sum_{d=1}^{d_k} q_{i,d}\, k_{j,d}$: it adds up $d_k$ separate products. If the query and key components are roughly independent with unit variance, each product contributes variance $\approx 1$, and variance adds, so the whole sum has variance $\approx d_k$. That means a typical score has magnitude around $\sqrt{d_k}$, and it grows as the head gets wider. Feed scores that large into softmax and it saturates: almost all the weight collapses onto a single token, and the gradient through softmax goes nearly flat, which stalls learning.

Dividing by $\sqrt{d_k}$ is exactly the fix: variance $d_k$ divided by $(\sqrt{d_k})^2 = d_k$ brings the variance back to $\approx 1$, independent of how wide the head is. So $\sqrt{d_k}$ isn’t arbitrary: it’s the standard deviation of the unscaled scores, the precise amount needed to renormalize them. For $d_k = 64$, that factor is $\sqrt{64} = 8$.

Finally, we apply the softmaxsoftmaxFunction that turns any vector into a probability distribution (positive, sums to 1) by exponentiating and normalizing.See in glossary → function across each row:

Softmax turns any row of real numbers into a probability distribution: every weight is between 0 and 1, and they sum to 1 across $j$. The high-scoring keys end up with most of the weight; the rest get squeezed toward zero.

Combining: weighted sum of values

We now have, for each token $i$, a row of attention weights $w_{i,1}, w_{i,2}, \ldots, w_{i,n}$ that sum to 1. These weights are how much of each other token to mix in. The remaining question is: mix in what?

This is where the third projection finally pays off. Recall we made $W_V x$ produce a value vector $v_j$ at every position. The queries and keys were used only to figure out the weights; they don’t appear in the output. The values are the payload: the actual content vector that gets sent forward when a token gets attended to.

So the new representation for token $i$ is constructed by taking every other token’s value vector, scaling it by how much $i$ cares about that token, and adding them all up:

Each $v_j$ is a $d_v$-dimensional vector, and so is $y_i$. The weights tell you the mixing ratios: if $w_{i,3} = 0.7$ and $w_{i,5} = 0.2$ and the rest are tiny, then 70 % of the resulting $y_i$ comes from token 3’s content, 20 % from token 5’s, and a small smear from everything else. In effect, every token gets to “pull in” content from the tokens it found relevant, weighted by how relevant.

That’s the entire operation. The three projections do three different jobs:

• $q_i$: “what am I asking for?”
• $k_j$: “what kind of thing am I?” (used to compute relevance against queries)
• $v_j$: “if you pick me, here is the content I’ll contribute”

You can think of it as a soft, differentiable dictionary lookup: query meets keys to decide who; values are the what.

In matrix form

Doing this for every token $i$ at once, and lining everything up into matrices, gives you the famous formula from “Attention Is All You Need”:

Reading left to right: compute all pairwise scores ($Q K^\top$), scale by $\sqrt{d_k}$, softmax each row, then multiply the resulting weight matrix by $V$, and because matrix multiplication is “weighted sums of rows,” that last multiply is exactly the per-token mixing we just did, executed for all tokens in parallel. We call this scaled dot-product attentionscaled dot-product attentionsoftmax(QKᵀ / √d_k) · V — the canonical attention formula from “Attention is All You Need”.See in glossary →. It is the centerpiece of the standard transformer architectures, including the decoder-only LLMs covered in this series.

The causal mask

There’s one wrinkle specific to generation. When the model produces text autoregressively, position $i$ must not be allowed to attend to positions in the future ($j > i$). Otherwise the model would “cheat” by peeking at tokens that come later. To prevent this, we set those scores to $-\infty$ before the softmax, which makes $\exp(-\infty) = 0$ and zeros them out cleanly.

This is the causal mask. It’s why the heatmap below is triangular: the lower-left half is computed, the upper-right is forbidden.

Try it

Below is an interactive attention map for our running example: the model has read “The cat sat on the soft,” and the output at the final position, soft, is used to predict the next token. Pick a head profile (one of a few canonical attention patterns), then click any row to see how that token’s weights are constructed. By default we show the final position. You’ll see the raw $Q\cdot K^\top$ scores, the softmax that turns them into a probability distribution, and the resulting weighted-value combination.

A few things to look for as you click around:

• Pattern A attends mostly to the previous token. For the final position, that’s “the”, an example of a head using local context.
• Pattern B matches content: the “soft” position pulls on “cat,” and the two “the”s lock together.
• Pattern C spreads attention evenly over all preceding tokens, a kind of “summarizer” that mixes in broad context.
• Pattern D places much of its probability mass on the first token. This type of “attention sink” has been observed in trained models, often near initial tokens. It is a useful pattern to recognize, but the weights alone do not establish why a head uses it.

Notice the upper-right triangle is always masked: position 2 (“sat”) can attend to positions 0, 1, 2 but not 3+. That’s the causal mask in action.

Each of these is one plausible thing a single attention computation could learn, but a real model needs to track many such patterns at once: tense, subject-verb agreement, coreferencecoreferenceWhen two words in a text refer to the same thing. In "Marie went home because she was tired," the pronoun "she" co-refers to "Marie." Resolving coreference — figuring out which earlier mention a pronoun, "this", "the company", etc. points back to — is one of the relationships transformer heads learn to track during training.See in glossary →, sentence boundaries, you name it. The mechanism that lets one layer run many independent attention patterns side by side is called multi-head attention, and it’s the next section.

What just happened, at the level of vectors

The end result of one attention layer is that every position’s vector has been updated to incorporate information from positions it considered relevant. The “soft” position’s vector can now contain some “cat” flavor and some information from earlier context, weighted by how much the model finds each relevant. The model gets to learn what features matter.

One catch: what we have described so far is a single attention head. Its attention distribution can vary across positions and inputs, but it is built from one set of query, key, and value projections and has only one learned subspace. Real language needs several kinds of relationships to be available at once: tense, subjects, references, sentence boundaries, and more. The fix is to run several attention heads side by side, each with its own $W_Q$, $W_K$, and $W_V$: multi-head attention, the next section.