Attention Is All You Need
Lesson 1Why attention? Moving beyond RNNs
Sequential processing limits scale, attention fixes that
Why attention? The three problems with RNNs and LSTMs
Before transformers, RNNs and LSTMs were the main architectures for sequences. They read text one token at a time, like turning pages in order: each new word updates a hidden state that is supposed to compress everything seen so far.
That story is intuitive, but it collides with modern hardware and long contexts. The rest of this course replaces recurrence with attention; this lesson names three concrete problems so the design choices later feel inevitable.
For an interactive walkthrough of the full transformer story (including this section), see the companion material at
Attention is All You Need, Lex AI
.
The big picture (three pain points)
| Problem | What goes wrong | What we want instead |
|---|---|---|
| 1 · No parallelization | Step t waits on step t − 1; GPUs sit idle along the time axis. | Process many positions at once where the task allows it. |
| 2 · Long-range paths | Early information reaches late tokens only by passing through many hops of the same hidden bottleneck. | Direct routes between distant words. |
| 3 · Backprop through time | Gradients traverse the whole unrolled chain, often vanish or explode. | Shorter, stable paths for learning signals. |
Vaswani et al. (2017), "Attention Is All You Need" is exactly the paper that proposed self-attention as the workhorse that addresses these issues together, we unpack the mechanics in the lessons ahead.
Predict before you look
In a 100-word sentence, how many hidden-state hops must the meaning of word 1 survive to influence word 100 in an RNN? And in an attention layer? Hold a guess, then read the picture below.
All three problems are really one shape. Here it is in a single diagram, the whole rest of this lesson is just unpacking the two rows:
The same short sentence. On top, the meaning of “The” has to relay through every hidden state to reach the end, and it dilutes on the way. Underneath, attention wires every word straight to every other in one hop, all at once. That is the whole motivation for the rest of this course.
Answer to the prediction: an RNN needs 99 hops (word 1 relays through every state to reach word 100); attention needs 1. The three sections below name why that difference matters for speed, memory, and training.
1 · Sequential processing = no parallelization
Each position must be processed after the previous one, like dominoes: token 10 cannot finish until tokens 1–9 have updated the hidden state.
Core tension
Massive GPU parallelism vs an algorithm that is inherently serial along time.
Toy example, sentence: The cat sat.
| Time step | Input token | State |
|---|---|---|
| 1 | "The" | → hidden state h₁ |
| 2 | "cat" | uses h₁ → h₂ |
| 3 | "sat" | uses h₂ → h₃ |
RNN unrolling (must run left → right)
2 · Long-range dependencies are hard
Information from an early token only reaches a late token by flowing through every intermediate hidden state. LSTMs (gates, cell state) ease this but do not erase it: over 50+ words, the signal is easy to dilute.
Linguistic example (subject–verb agreement across a long relative clause):
The keys, which were lying on the kitchen counter this morning before I left for work and forgot to pick up, are still there.
The model should keep plural "keys" aligned with plural "are" even though many words sit in between.
An RNN must carry that agreement signal through every step in the chain, there is no shortcut.
Feel it yourself. Drag the distance between two words and the memory kept per hop. Watch how quickly the early signal (and, in section 3, the gradient) fades in an RNN while attention holds a direct one-hop link:
17 hops later, only 3.4% of the early signal survives. The gradient shrinks by the same product.
3 · Weak gradient flow (BPTT)
Training unrolls the network over the sequence and runs backpropagation through time (BPTT). The gradient at an early timestep depends on a product of many Jacobian factors, one per hop.
Gradient path (conceptual)
∂L/∂h₁ ← ··· ← ∂L/∂h₁₀ ← ∂L/∂h₂₀ ← ··· ← ∂L/∂h₁₀₀
(each hop multiplies into the product)Vanishing
Factors shrink along the chain → early layers get almost no update.
Exploding
Factors grow → unstable updates unless you clip or tune carefully.
During backpropagation every gradient is multiplied by the activation's derivative. See what that does layer by layer.
σ'(z) = σ(z)(1−σ(z)) ≤ 0.25Sigmoid clamps every derivative to at most 0.25. Across 10 layers that gives 0.25¹⁰ ≈ 0.000001 — one millionth of the original gradient. Early layers receive essentially zero signal and stop learning entirely. This is why sigmoids were abandoned for hidden layers.
Mental model: conveyor belt vs direct wires
| RNN / LSTM | Transformer (preview) |
|---|---|
| One conveyor belt of hidden states | Many direct pairwise links per layer (attention weights) |
| Distance from word i to j = O(|i − j|) steps along time | Information can mix in one attention layer (subject to masks in decoders) |
| Whole-sequence summary squeezed into a fixed-size vector at each step | Each position keeps a vector; attention selects what to read |
That shift, parallelizable, direct paths, shorter gradient routes, is why transformers scaled to today's language models. Next we implement the first ingredient: dense token vectors (embeddings).
One sentence to remember
Recurrence makes every word relay through a single fading chain (serial, long paths, unstable gradients); attention gives every pair a direct one-hop wire computed in parallel. That is the top row versus the bottom row of the diagram above.
Check your understanding
1 / 6Why are RNNs and LSTMs fundamentally sequential during training?
Practice it yourself
Put numbers on why recurrence lost: path length, serial steps, and per-layer cost. Real Python, hidden tests, no setup.