The residual stream grows without LayerNorm

Background

Each block adds to the residual stream: x_ℓ = x_{ℓ-1} + f_ℓ(x_{ℓ-1}). With nothing normalising it, the activation magnitude tends to grow with depth — attention scores blow up, softmax saturates, training plateaus. Tracking the norm after each block makes the problem visible (and motivates LayerNorm next lesson).

Problem statement

Implement residual_stream_norms(x0, deltas) returning the L2 norm of the stream after the initial state and after each added delta.

Input

• x0 — initial residual stream (vector or matrix).
• deltas — list of block outputs f_ℓ(x_{ℓ-1}), each same shape as x0.

Output

Returns a list of float of length len(deltas) + 1: [‖x0‖, ‖x0+δ1‖, ‖x0+δ1+δ2‖, …] (L2 / Frobenius norm).

Examples

Example 1 — aligned deltas grow the norm

Input:  x0 = [1, 0], deltas = [[1, 0], [1, 0]]
Output: [1.0, 2.0, 3.0]

Example 2 — zero delta leaves the norm unchanged

Input:  x0 = [3, 4], deltas = [[0, 0]]
Output: [5.0, 5.0]

Constraints

• Start from x0; record ‖x0‖.
• For each delta: add it to the running stream, record the new norm.
• Use the L2 (Frobenius) norm; return floats, length len(deltas)+1.

Notes

• The accumulation is exactly why GPT-2 puts a LayerNorm in front of every sublayer — to rescale the stream so deep blocks see well-conditioned inputs.