Weight tying — the LM head reuses E

Background

GPT-2 ties weights: the same embedding matrix E ∈ ℝ^{V×C} that maps token ids to vectors is reused (transposed) to map the final hidden states back to vocabulary logits.

$$\text{logits} = H E^\top, \qquad H \in \mathbb{R}^{T \times C},\; E \in \mathbb{R}^{V \times C},\; \text{logits} \in \mathbb{R}^{T \times V}$$

So logits[t, v] = H[t] · E[v] — how strongly position t predicts token v. This saves ~38M parameters and tends to improve perplexity.

Problem statement

Implement tied_lm_head(H, E) returning the logits.

Input

• H — hidden states, shape (T, C).
• E — tied embedding matrix, shape (V, C).

Output

Returns an np.ndarray of shape (T, V): H @ E.T.

Examples

Example 1

Input:  H = [[1, 0]], E = [[1, 0], [0, 1], [1, 1]]
Output: [[1, 0, 1]]

Explanation: H[0]·E[0]=1, ·E[1]=0, ·E[2]=1.

Constraints

• Compute H @ E.T.
• Return shape (T, V).

Notes

• The same vector serves two roles: "what token i means" (a row of E on the way in) and "how much to predict token i" (the same row on the way out). Tying says those should be the same geometry.