How big is GPT-2's embedding table?

Background

GPT-2's input layer has two learned tables:

• Token embeddings wte: vocab_size × n_embd.
• Positional embeddings wpe: block_size × n_embd.

For GPT-2 small (vocab_size=50257, block_size=1024, n_embd=768) these alone account for ~39M of the model's 124M parameters — about a third.

Problem statement

Implement embedding_params(vocab_size, block_size, n_embd) returning (token_params, pos_params, total).

Input

• vocab_size — BPE vocabulary size.
• block_size — maximum context length.
• n_embd — embedding dimension.

Output

Returns (token_params, pos_params, total):

• token_params = vocab_size * n_embd,
• pos_params = block_size * n_embd,
• total = token_params + pos_params.

Examples

Example 1 — GPT-2 small

Input:  vocab_size = 50257, block_size = 1024, n_embd = 768
Output: (38597376, 786432, 39383808)

Example 2 — tiny config

Input:  vocab_size = 10, block_size = 4, n_embd = 2
Output: (20, 8, 28)

Constraints

• token_params = vocab_size * n_embd; pos_params = block_size * n_embd.
• total is their sum.
• Return (int, int, int).

Notes

• The token table dominates because the vocabulary (50k) dwarfs the context length (1k) — most embedding parameters are one row per BPE token.