Problem with RNNs and LSTMs

Why GPT-2 is a transformer, not an RNN

Before we write a single line of PyTorch, let's answer the obvious question: why isn't GPT-2 an RNN? Recurrent networks dominated language modelling for almost a decade, LSTMs powered Google Translate, speech recognition, and the first wave of seq2seq systems. Then, between 2017 and 2019, the entire field switched to transformers. Three concrete failures pushed it over.

This course is going to build a decoder-only transformer from scratch. To understand why we make every architectural choice we make, embeddings, causal masking, multi-head attention, residual connections, you need to know what we're walking away from and why we couldn't just keep scaling the old approach.

Set up the project

Every lesson in this course adds one or more files to a single Python package, nanogpt-from-scratch/. By Lesson 10 you'll run python train.py and watch your own GPT-2-small train on TinyShakespeare in about 10 minutes on a Mac. The course is the codebase.

1. Get the scaffold

The companion scaffold lives at /code/nanogpt-from-scratch/. It contains stub files for every module we'll fill in across the course. Either download the files individually (each lesson tells you which file to open) or pull the whole directory:

mkdir nanogpt-from-scratch && cd nanogpt-from-scratch
# fetch the README, pyproject.toml, and the empty .py stubs from /code/nanogpt-from-scratch/

2. Install dependencies

python -m venv .venv && source .venv/bin/activate
pip install -e .

That installs three packages: torch (the model), tiktoken (GPT-2's BPE tokenizer, used in Lesson 2), and numpy. Tested on Python 3.10+. CPU works fine for everything up through Lesson 9; for Lesson 10's training run, an Apple-Silicon MPS device or a CUDA GPU will speed things up ~5×.

3. Write config.py

Open the empty config.py and paste this in. These defaults are exactly the GPT-2 small architecture from Radford et al. (2019), 124M parameters, the smallest of the four GPT-2 models OpenAI released:

# nanogpt-from-scratch/config.py
 
from dataclasses import dataclass
 
 
@dataclass
class GPTConfig:
    vocab_size: int = 50257   # GPT-2 BPE vocabulary
    n_embd: int = 768         # embedding / hidden dimension
    n_head: int = 12          # number of attention heads
    n_layer: int = 12         # number of transformer blocks
    block_size: int = 1024    # maximum context length
    dropout: float = 0.0      # 0.0 for pretraining, ~0.1 for finetuning
 
    @property
    def head_size(self) -> int:
        assert self.n_embd % self.n_head == 0, "n_embd must divide n_head"
        return self.n_embd // self.n_head

Why a dataclass and not a dict or a config file? Two reasons: autocomplete (your editor shows every field while you type config.), and single source of truth (every other module, attention.py, blocks.py, model.py, takes a GPTConfig and reads config.n_embd, config.n_head, etc.). Change one number here, the whole model resizes. Karpathy's nanoGPT uses the same pattern, and so does the original OpenAI GPT-2 codebase.

That's the whole project setup. Now the rest of this lesson is why we're going to fill those stubs with attention layers and not LSTM cells. Three concrete failures pushed the field off RNNs.

Failure 1, Sequential processing kills GPU parallelism

An RNN computes its hidden state one token at a time:

Notice the dependency: h_t cannot be computed until h_{t-1} is finished. To process a 1024-token sequence, the GPU must execute 1024 sequential matrix multiplies, one after another, no shortcuts. Modern GPUs have ~10,000+ cores sitting idle because the math forbids using them in parallel.

Transformers, by contrast, compute every token's representation simultaneously through self-attention. A 1024-token sequence becomes a single batched matrix multiply. On the same hardware, transformer training throughput is 10–100× higher than an equivalent RNN. Without this single change, GPT-2's 1.5B-parameter scale would not be economically possible.

This is the silent killer. Even before we get to gradients or context, RNNs lose by virtue of not being able to use the hardware they're running on.

Failure 2, Vanishing gradients across long sequences

When you backpropagate through an RNN unrolled across T timesteps, the gradient at step 0 is a product of T Jacobians:

Each factor in that product is roughly tanh'(·) · W_h. Since tanh' ∈ (0, 1], repeated multiplication shrinks the gradient exponentially. Over a few hundred timesteps, the gradient at early tokens collapses to numerical noise, the network cannot learn dependencies that span more than ~10–20 tokens.

The visualization below makes this concrete. Pick an activation, pick a depth, watch the gradient magnitude at each layer. The same dynamic that wrecks deep MLPs wrecks RNNs unrolled across long sequences:

LSTMs were specifically designed to mitigate this, the cell state $c_t = f_t \odot c_{t-1} + i_t \odot g_t$ provides a near-identity gradient path when the forget gate $f_t \approx 1$. This works for moderate sequence lengths (50–200 tokens) but it doesn't scale: at 1000+ tokens, even LSTMs degrade. A model trained to predict the next token in a Wikipedia article can't reliably remember the article's title.

Transformers sidestep this entirely. Self-attention gives every token a direct path to every other token, gradient flow from token 1024 to token 1 traverses one attention layer, not 1024 sequential cells. The longest gradient path in a transformer is O(layers), not O(sequence length).

Failure 3, Information bottlenecked through a fixed-size hidden state

This one is subtler but architecturally fatal. An RNN compresses everything it has seen so far into a single fixed-dimensional hidden state h_t ∈ ℝ^d. Whether it has read 10 tokens or 10,000, all of that context must be squeezed into the same d numbers and passed forward.

Think about what this means for a language model. To predict the next word in:

"The agreement, signed in 1997 by representatives of nine member states after eighteen months of fractious negotiation in Brussels, ..."

…the model needs to reach back ~25 tokens to recall what was signed (the agreement) and another 8 tokens to recall where it was signed (Brussels). An RNN is trying to keep both of those facts alive in its hidden state while also tracking grammar, current clause structure, and every other detail that mattered along the way. Information competes for room in a fixed-size vector.

Self-attention removes the bottleneck completely. Every token in the context is stored separately, there is no compression step. When the decoder needs to attend back to "agreement," it does so directly, regardless of how many tokens have passed. The transformer's effective memory grows linearly with context length; the RNN's stays fixed at d.

What this gives us

These three failures are why GPT-2, GPT-3, and every modern LLM are transformers. The decoder-only architecture we're about to build:

• Processes the whole sequence in parallel, exploits GPU compute fully
• Provides direct gradient paths between any two positions, long-range dependencies are learnable
• Stores every token explicitly in the attention KV cache, no fixed-size bottleneck

Each lesson in this course corresponds to one component that delivers one of these properties. By Lesson 10, you'll have an end-to-end PyTorch implementation of a small GPT-2 you can train on your own data.

What we keep from the RNN era

Not everything the RNN community learned is obsolete. Two ideas in particular survived the transition:

• Tokenization and embeddings, turning discrete symbols into dense vectors is still the first step of every LM. Lesson 2.
• Cross-entropy loss on next-token prediction, the training objective is unchanged. The architecture changed; the task did not.

The transformer is not a wholesale rejection of what came before. It's a targeted fix to three specific bottlenecks that made RNNs a dead end at scale.

What's next

config.py is in place, that's the whole shape of GPT-2 small in 12 lines of code. Lesson 2 opens up tokenizer.py and embeddings.py: how to turn a string of English into a sequence of integers (BPE tokenization), and how to turn each integer into a 768-dimensional vector the model can actually compute on. About a third of GPT-2's 124M parameters live in the embedding table, we'll see exactly where they come from.

Practice it yourself

Pin down the GPT-2 config math and the RNN bottleneck this course walks away from. Real Python, hidden tests, no setup.