Embeddings.forward — token + position (batched)

Background

The nanoGPT Embeddings layer sums token and positional embeddings for a batch of sequences. Token embeddings are looked up per id (B, T, C); positional embeddings are (T, C) and broadcast across the batch:

$$y_{b,t,c} = E[\text{idx}_{b,t}, c] + P[t, c]$$

Problem statement

Implement embeddings_forward(idx, E, P) returning the (B, T, C) input tensor.

Input

• idx — token ids, shape (B, T).
• E — token table, shape (V, C).
• P — positional table, shape (block_size, C) with block_size ≥ T.

Output

Returns an np.ndarray of shape (B, T, C): E[idx] + P[:T] (positional broadcasts over the batch).

Examples

Example 1

Input:  idx = [[0, 2]],
        E = [[1, 1], [2, 2], [3, 3]],
        P = [[10, 10], [20, 20]]
Output: [[[11, 11], [23, 23]]]

Explanation: E[0]+P[0]=[11,11], E[2]+P[1]=[3,3]+[20,20]=[23,23].

Constraints

• tok = E[idx] → (B, T, C); pos = P[:T] → (T, C).
• Return tok + pos (broadcast over batch).
• Shape (B, T, C).

Notes

• This is what makes the same token at two positions produce different vectors — the fix for attention's permutation-blindness.