The 4× MLP hidden dimension

Background

GPT-2's feed-forward block widens the embedding to a hidden dimension 4× larger, applies a non-linearity, then projects back:

$$\text{Linear}(C, 4C) \to \text{GELU} \to \text{Linear}(4C, C)$$

The 4× expansion (d_ff = 4·C) has been the default since the 2017 Transformer — wide enough for rich per-token feature mixing, not so wide it wastes parameters.

Problem statement

Implement mlp_hidden_dim(n_embd, expansion=4) returning the hidden width.

Input

• n_embd — embedding dim C.
• expansion — width multiplier (default 4).

Output

Returns an int: expansion * n_embd.

Examples

Example 1 — GPT-2 small

Input:  n_embd = 768
Output: 3072

Example 2 — 8× variant

Input:  n_embd = 512, expansion = 8
Output: 4096

Constraints

• Return expansion * n_embd.
• Default expansion = 4.

Notes

• The hidden layer projects onto many feature directions before the non-linearity recombines them — the per-token "compute" tier attention can't provide.