Fixed vs growing memory (the RNN bottleneck)

Background

The RNN's third failure: it crams all context into one fixed-size hidden state h ∈ ℝ^d, so its effective memory is d no matter how long the sequence. A transformer stores every token's representation separately, so its memory grows with context length:

• RNN: d (fixed)
• Transformer: context_len · d (one d-vector per token)

Problem statement

Implement effective_memory(arch, context_len, d) returning the number of stored activations.

Input

• arch — "rnn" or "transformer".
• context_len — int, number of tokens seen.
• d — int, hidden dimension.

Output

Returns an int: d for "rnn", context_len * d for "transformer".

Examples

Example 1 — RNN is fixed

Input:  arch = "rnn", context_len = 1000, d = 768
Output: 768

Example 2 — transformer grows

Input:  arch = "transformer", context_len = 1000, d = 768
Output: 768000

Constraints

• "rnn" → d (ignores context_len); "transformer" → context_len * d.
• Return a plain int.

Notes

• This is why an RNN "forgets the article's title": old facts compete for room in the same d numbers, while a transformer keeps them all addressable.