Head dimension d_k = d_model / h

Background

Multi-head attention splits the model width across heads so that concatenating them restores d_model. Each head works in a smaller space of width:

$$d_k = \frac{d_{\text{model}}}{h}$$

For the paper's d_model = 512, h = 8 heads → d_k = 64.

Problem statement

Implement head_dim(d_model, h) returning the per-head dimension.

Input

• d_model — int, model width.
• h — int, number of heads (divides d_model).

Output

Returns an int: d_model // h.

Examples

Example 1 — the paper

Input:  d_model = 512, h = 8
Output: 64

Example 2 — GPT-2 small

Input:  d_model = 768, h = 12
Output: 64

Constraints

• Return integer division d_model // h.

Notes

• This split is what keeps total compute roughly constant as you add heads — h heads of width d_model/h cost about the same as one head of width d_model.