What Is Attention? How a Word Decides Which Other Words Matter
The perceptron article ended with "plus the attention machinery around them". And in Putting It Together, the layer tower had a mysterious label: some layers were "full attention". Time to open that box.
A perceptron looks at one token's numbers at a time. But language doesn't work one word at a time. In "The animal didn't cross the street because it was too tired", you can't understand "tired" without knowing who was tired. Attention is the mechanism that lets each word look back at the earlier words and pull in what it needs.
It's the core idea of the Transformer, from the 2017 paper Attention Is All You Need, and every model in this series is built on it.
The idea in one paragraph
Every word produces three vectors, each one a perceptron-style multiplication by learned weights:
- a query: "what am I looking for?"
- a key: "what do I contain?"
- a value: "what do I pass on if you pick me?"
Each word compares its query with the key of every earlier word (a dot product: multiply and add), turns those scores into weights that add up to 100% (softmax, the same step that turned 248,320 scores into probabilities in the last article), and takes the weighted mix of the values. The word's new meaning is a blend of the words it paid attention to.
weights = softmax( query · keysᵀ / √d )
output = weights · values
That's it. The rest is doing it many times in parallel: each layer has several heads, each with its own query, key and value weights, so each head can look for a different kind of relationship.
Try it
Part 1 is a toy example with two-number vectors you can steer. Part 2 is the real thing: attention weights I read out of Qwen2.5-0.5B (Apache 2.0, 24 layers × 14 heads = 336 heads) on my Mac. Part 3 shows why attention gets expensive.
The experiment: "tired" vs "wide"
When Google introduced the Transformer in 2017, its blog post used a famous pair of sentences: the animal didn't cross the street because it was too tired, or too wide. A human knows instantly that "it" is the animal in the first and the street in the second.
I fed both sentences to Qwen2.5-0.5B and read the attention weights of every head in every layer.
First, an honest twist. Models like Qwen and GPT write left to right, and each word can only look back. So when the model reads "it", it hasn't seen "tired" or "wide" yet. I checked: the attention from "it" is identical in both sentences, in every head. (Google's example came from a model that reads in both directions.) For a left-to-right model, the decision happens later, at the word that settles it.
At the last word, the meaning flips. In one head (layer 6, head 6):
| Last word | attention to "animal" | attention to "street" |
|---|---|---|
| "…too tired" | 73.2% | 5.1% |
| "…too wide" | 24.4% | 27.0% |
"Tired" goes straight to the animal. "Wide" pulls a big share toward the street. And it isn't one lucky head: 34 of the 336 heads favour "animal" when the sentence ends in "tired" and switch to favour "street" when it ends in "wide". The grid in part 2 shows every head at once.
The surprise: the first word gets most of the attention
Averaged over every head, and every word after the first, 61% of all attention goes to the very first token, "The". That looks like a bug, but it's a known behaviour called an attention sink. Softmax forces each head's weights to add up to 100%, even when a head has nothing useful to look at. Models learn to park that unused attention on the first token, which every later word can see. Researchers described this in 2023 (Efficient Streaming Language Models with Attention Sinks), and they used it to keep long conversations stable.
You can see it in part 2: pick almost any head, and "The" takes a big bar. The heads that do something interesting are the ones that move attention away from the sink, toward words like "animal".
Why attention is expensive
Each word scores every earlier word. For a text of n tokens that's about n²/2 scores, per head, per layer:
| Context | scores per head, per layer |
|---|---|
| 1,024 tokens | about 525,000 |
| 131,072 tokens | about 8.6 billion |
Double the length and the work roughly quadruples. That's the main reason long context windows are expensive, and why newer models look for cheaper alternatives. Remember the tower in the last article: Qwen3.5 uses "linear attention", which keeps a running summary instead of comparing every pair, in 18 of its 24 layers, and full attention in only 6.
How much to trust this
- One small model and one sentence pair. Qwen2.5-0.5B, chosen because it uses standard attention I could read cleanly. Other models, or other sentences, will spread the work across different heads.
- Attention weights aren't the whole story. They show where a head looks, not what it does with what it finds; the values and the layers after it matter too. They're a window into the model, not a full explanation.
- The toy in part 1 uses made-up two-number vectors. The real model's queries and keys have 64 numbers per head, and nobody chose what they mean: training did.
The one-line mental model
Attention lets every word ask a question of the words before it, score each one, and blend in the answers it likes best. Do that in hundreds of heads at once, and "tired" can find the animal while "wide" finds the street.