Where loss starts: log(V)

Background

A freshly-initialised language model has no idea what comes next, so its predicted distribution is roughly uniform over the V-token vocabulary. The cross-entropy of a uniform guess is:

$$\mathcal{L}_0 = -\log\frac{1}{V} = \log V$$

For GPT-2's V = 50257, that's ≈ 10.83 nats. Much above this at step 0 signals an init bug; much below signals label leakage (e.g. a missing causal mask).

Problem statement

Implement random_init_loss(vocab_size) returning the expected starting cross-entropy.

Input

• vocab_size — int, the vocabulary size V.

Output

Returns a float: ln(vocab_size) (natural log).

Examples

Example 1 — GPT-2

Input:  vocab_size = 50257
Output: 10.824637...

Example 2

Input:  vocab_size = 2
Output: 0.693147...

Constraints

• Return math.log(vocab_size) (natural log).

Notes

• This single number is the cheapest sanity check on a fresh training run — print step-0 loss and compare to log V.