Every large language model — GPT, Claude, Gemini — is built from one operation repeated hundreds of times: attention. Each word looks at every other word, decides how relevant each is, and pulls in a weighted blend of their meaning. That's it. The 2017 paper was literally titled "Attention Is All You Need." Hover a row to see which words a token attends to.
how a word decides what to look at
Each token starts as a vector (its embedding + position). Attention projects every token into three vectors via learned matrices: a query Q ("what am I looking for?"), a key K ("what do I offer?"), and a value V ("what I'll contribute"). Then:
Attention(Q,K,V) = softmax(Q·Kᵀ / √d) · V
Step by step:
1. Scores = Q·Kᵀ — every query dotted with every key, a full grid of "how much should token i attend to token j." High dot product = aligned meaning.
2. Divide by √d (the dimension) to keep gradients stable, then softmax each row so the weights are positive and sum to 1 — a probability distribution over which tokens to read from.
3. The output for each token is the weighted average of all value vectors, using those attention weights. Token i becomes a blend of everyone it attended to.
The grid above is the attention matrix — row i, column j is how much token i attends to token j (brighter = more). This is exactly what you see in "attention maps" of real models: pronouns attend to their referents, verbs to their subjects, the start token soaks up the rest.
Causal mask: in a generative model, a token may only attend to earlier tokens (you can't see the future you're predicting). That zeroes the upper triangle — toggle it above and watch the grid become lower-triangular.
Why it changed everything: unlike RNNs, attention connects any two positions in one step, regardless of distance, and the whole grid computes in parallel — perfect for GPUs. Stack many attention "heads" (each learning a different relation) and many layers, scale to trillions of parameters and the open internet, and you get models that write, reason, and code. Temperature here mimics how sharp or diffuse attention is — low temperature makes each token fixate on one other; high temperature spreads it evenly. The same softmax governs how the model finally picks its next word.