The position-wise feed-forward network

Background

The second sublayer of every encoder block is a position-wise feed-forward network — the same two-layer MLP applied independently to each position:

$$\text{FFN}(x) = \max(0,\, xW_1 + b_1)\,W_2 + b_2$$

It expands to a wider hidden size d_ff (2048 in the paper for d_model=512), applies ReLU, then projects back to d_model.

Problem statement

Implement ffn(x, W1, b1, W2, b2) returning the position-wise FFN output.

Input

• x — shape (n, d_model).
• W1 (d_model, d_ff), b1 (d_ff,), W2 (d_ff, d_model), b2 (d_model,).

Output

Returns an np.ndarray of shape (n, d_model).

Examples

Example 1

Input:  x = [[1, -1]], W1 = [[1, 0], [0, 1]], b1 = [0, 0], W2 = [[1, 0], [0, 1]], b2 = [0, 0]
Output: [[1, 0]]

Explanation: x@W1 = [1, -1]; ReLU → [1, 0]; @W2 + b2 → [1, 0].

Constraints

• Compute relu(x @ W1 + b1) @ W2 + b2.
• Return shape (n, d_model).

Notes

• "Position-wise" means the same weights are applied to every token independently — it mixes features, not positions (attention already mixed positions).