The input layer — token lookup + positional

Background

The transformer's input matrix Z is built by looking up each token's embedding and adding the positional vector for its slot:

$$Z_t = E[\text{ids}_t] + P_t, \qquad t = 0, \ldots, L-1$$

where E is the (V, d) token table and P is the (L_max, d) learned positional table (use its first L rows).

Problem statement

Implement input_embeddings(token_ids, E, P) returning the (L, d) input matrix Z.

Input

• token_ids — list of L integer token IDs.
• E — token embedding table, shape (V, d).
• P — positional table, shape (L_max, d) with L_max ≥ L.

Output

Returns an np.ndarray of shape (L, d): E[token_ids] + P[:L].

Examples

Example 1

Input:  token_ids = [0, 2],
        E = [[1, 1], [2, 2], [3, 3]],
        P = [[0, 0], [10, 10], [20, 20]]
Output: [[1, 1], [13, 13]]

Explanation: row 0 = E[0] + P[0] = [1,1]; row 1 = E[2] + P[1] = [3,3] + [10,10] = [13,13].

Constraints

• Gather E[token_ids] (shape (L, d)).
• Add the first L rows of P.
• Return shape (L, d).

Notes

• The token row depends only on which word; the positional row depends only on which slot. Their sum is the representation attention starts from — before any Q/K/V projection.