One idea.
Six generations.

A standard encoder–decoder RNN reads the source sentence token by token, and by the final step the encoder's last hidden state is expected to hold everything — every word, every dependency, every nuance — squeezed into one vector of fixed size. It doesn't matter if the sentence is five words or fifty; the vector stays the same size. The decoder then generates the entire translation from that single vector alone, never looking back at the source again.

Try it — quality vs. sentence length

Drag the slider to see the pattern the original paper observed: a plain encoder–decoder degrades as sentences get longer, while a model that can look back at every source word holds steady.

Bahdanau's fix — let the decoder look back

Instead of discarding the encoder's hidden states after producing the final summary, keep all of them — \(h_1, h_2, \dots, h_{T_x}\) — and let the decoder consult all of them at every single output step, deciding fresh each time which ones matter most.

Because the scoring function concatenates \(s_{t-1}\) and \(h_j\) and pushes them through a learned \(\tanh\) layer rather than a dot product, this is called additive attention. \(c_t\) is recomputed fresh at every single decoding step, so the model attends to a different part of the source sentence for each output word — and because the whole thing is differentiable, it trains end to end with ordinary backpropagation, no separate alignment step required.

For every token, learn three separate projections of its embedding: a query (what am I looking for), a key (what do I offer, for matching purposes), and a value (what content do I actually contribute once selected). This is the key generalisation over Bahdanau, who reused one encoder state for both matching and content — separating key and value gives the model strictly more room to learn.

A useful mental model borrowed from information retrieval: think of it as a soft database lookup. Every token throws a query into the room ("who has information relevant to me?"), every token also holds up a key as an advertisement of what it contains, and matching a query against a key produces a relevance score. Unlike a real database, nothing is retrieved all-or-nothing — every token's value gets pulled in, just weighted by how relevant its key turned out to be.

\(X\) is the matrix of input embeddings (one row per token), and \(W_Q, W_K, W_V\) are learned weight matrices — the only parameters in a single attention head. Everything downstream is deterministic matrix arithmetic on \(Q\), \(K\), \(V\).

Block diagram — the full pipeline, one head

The \(\sqrt{d_k}\) scaling matters more than it looks: without it, dot products grow large as dimensionality increases, pushing softmax into a saturated region with near-zero gradients — the network stops learning from those positions. \(M\) is the optional mask matrix from the dashed box above — all zeros for bidirectional attention, or the causal pattern covered further down.

Block diagram — splitting into multiple heads

In practice you never run just one attention function. \(d_{model}\) is split across \(h\) heads, each with its own \(W_Q, W_K, W_V\) and a smaller dimension \(d_k = d_{model}/h\) — so the total compute stays roughly the same as one big head, but the model gets \(h\) independent "lenses" on the sequence instead of one averaged view.

Each head gets its own learned projections, so different heads can specialise — empirically, some track syntax, some track coreference, some track raw positional distance. Concatenating and projecting the results gives the model several independent "lenses" on the same sequence at once, instead of one averaged view.

Try it — click a token to see what it attends to

Two flavours of self-attention — and what each can see

Adding \(-\infty\) before softmax is what makes this work: \(e^{-\infty} = 0\), so those positions get exactly zero weight — not "very small," genuinely zero. That's what guarantees a decoder-only model can never leak information from a future token during training.

This is exactly what Bahdanau attention was doing conceptually — reformulated with learned Q/K/V projections and computed in the standard transformer attention form instead of an additive scoring MLP. In the original encoder–decoder transformer, every decoder layer runs masked self-attention over its own tokens first, then cross-attends into the encoder's final output — that second step is the only place source and target sequences actually connect.

Where cross-attention shows up

Try it — which pairs actually get computed?

Each cell is a (query, key) pair. Full attention lights up the whole grid. Everything below is a strategy for lighting up far fewer cells while still capturing what matters.

The methods

Block diagram — why the cache grows every single step

Autoregressive generation produces one token at a time, and every new token needs its own K and V computed and appended to what's already stored — recomputing K/V for every earlier token, every step, would be wasteful, so instead the cache just accumulates.

Head-sharing patterns, side by side

All four, compared directly

Block diagram — how MLA actually compresses the cache

GQA and MQA save memory by literally storing fewer K/V vectors. MLA takes a different approach entirely: it never stores per-head K/V at all. Instead, each token is compressed down into one small shared latent vector — that's the only thing written to the cache — and the full per-head K and V are reconstructed from it on the fly, fresh, every time attention is computed.

Only \(c_t\) — the compressed latent — ever touches the KV cache. The per-head \(K_i, V_i\) are recomputed from it inside every attention call and thrown away immediately after, so the cache footprint depends on the latent dimension \(d_c\) rather than on \(h \times d_h\) at all. DeepSeek-V2's reported result — a far smaller cache with quality matching or beating full MHA — is what makes this the most aggressive entry in the family, at the cost of noticeably more implementation complexity than GQA.

Try it — how much cache does grouping actually save?

A GPU has two very different kinds of memory: HBM (high-bandwidth memory — tens of gigabytes, but slow to access) and SRAM (on-chip cache — tiny, maybe 20MB, but extremely fast). Standard attention computes the full n×n score matrix and writes it out to HBM, then reads it back for softmax, then reads it again for the final matmul with V — several slow round trips to the big, slow memory for something that's only ever used briefly. FlashAttention never materialises that full matrix in HBM at all.

The trick that makes tiling possible is online softmax: softmax normally needs the full row of scores before it can normalise, but there's a numerically stable running-update formula that lets you process the row in small chunks, updating a running max and running sum as you go, and rescaling the accumulated output so far each time a new chunk arrives. That's what lets attention be computed block-by-block without ever holding the whole row in memory at once.