Add & Norm (post-norm residual)

Background

Every encoder sublayer is wrapped in Add & Norm: add the residual, then LayerNorm (the paper's post-norm order):

$$\text{out} = \text{LayerNorm}(x + \text{Sublayer}(x))$$

LayerNorm normalises each row over its d features to zero mean / unit variance, then scales and shifts:

$$\text{LN}(h)_j = \frac{h_j - \mu}{\sqrt{\sigma^2 + \varepsilon}}\,\gamma_j + \beta_j$$

Problem statement

Implement add_and_norm(x, sublayer_out, gamma, beta, eps=1e-5) returning LayerNorm(x + sublayer_out).

Input

• x — shape (n, d): the sublayer input (residual).
• sublayer_out — shape (n, d): the sublayer's output.
• gamma, beta — shape (d,): LayerNorm scale and shift.
• eps — float, numerical stability.

Output

Returns an np.ndarray of shape (n, d).

Examples

Example 1

Input:  x = [[1, 2]], sublayer_out = [[1, 2]], gamma = [1, 1], beta = [0, 0]
Output: [[-1, 1]]

Explanation: residual [2, 4]; mean 3, variance 1; normalised [-1, 1]; γ=1, β=0 leave it unchanged.

Constraints

• Compute the residual x + sublayer_out first.
• LayerNorm over the last axis (per row): subtract the row mean, divide by sqrt(var + eps), scale by gamma, shift by beta. Use population variance (mean of squared deviations).
• Return shape (n, d).

Notes

• Post-norm (norm after the add) is the original paper's choice; many modern models use pre-norm for deeper stacks.