Why one-hot inputs blow up memory

Background

Before learned embeddings, the naive input is a one-hot tensor of shape (B, T, V) — one giant sparse vector per token. Its size in bytes is:

$$\text{bytes} = B \cdot T \cdot V \cdot \text{bytes\_per}$$

For a batch of GPT-2 inputs (B=32, T=1024, V=50257, fp32 = 4 bytes) this is ~6.6 GB — per batch, before any matmul.

Problem statement

Implement onehot_memory(B, T, V, bytes_per=4) returning the byte size of the one-hot input tensor.

Input

• B — batch size, T — sequence length, V — vocabulary size.
• bytes_per — bytes per element (default 4 for fp32).

Output

Returns an int: B * T * V * bytes_per.

Examples

Example 1 — GPT-2 batch

Input:  B = 32, T = 1024, V = 50257, bytes_per = 4
Output: 6587285504

(~6.6 GB.)

Example 2 — tiny

Input:  B = 1, T = 1, V = 10
Output: 40

Constraints

• Return B * T * V * bytes_per.
• Default bytes_per = 4.

Notes

• A learned embedding replaces the V-wide one-hot with a dense C-wide vector (768 for GPT-2), so the input becomes (B, T, C) — ~65× smaller, and it carries similarity structure for free.