Backprop — all four gradients for one hidden layer

Background

The full backward pass for the canonical $i\to h\to o$ net (sigmoid everywhere, MSE loss) is four moves. Given the cached forward values and W2:

$$\boldsymbol{\delta}_2 = (\mathbf{a}_2 - \mathbf{y})\odot \mathbf{a}_2(1-\mathbf{a}_2), \qquad \boldsymbol{\delta}_1 = (W_2\,\boldsymbol{\delta}_2)\odot \mathbf{a}_1(1-\mathbf{a}_1)$$ $$\frac{\partial L}{\partial W_2} = \mathbf{a}_1\boldsymbol{\delta}_2^{\top},\;\; \frac{\partial L}{\partial \mathbf{b}_2} = \boldsymbol{\delta}_2,\;\; \frac{\partial L}{\partial W_1} = \mathbf{x}\boldsymbol{\delta}_1^{\top},\;\; \frac{\partial L}{\partial \mathbf{b}_1} = \boldsymbol{\delta}_1$$

Here W2 has shape (h, o), so W2 @ delta2 is the (h,) back-propagated error.

Problem statement

Implement backprop(x, y, a1, a2, W2) returning (dW1, db1, dW2, db2).

Input

• x — input vector (i,).
• y — true label (o,).
• a1 — cached hidden activation (h,).
• a2 — cached prediction (o,).
• W2 — output weights, shape (h, o).

Output

Returns (dW1, db1, dW2, db2) with shapes (i, h), (h,), (h, o), (o,) — each matching its parameter.

Examples

Example

Input:  x = [0.5, -0.5], y = [1.0], a1 = [0.6, 0.4], a2 = [0.7], W2 = [[0.5], [-0.5]]
Output: dW2 = [[-0.0378], [-0.0252]], db2 = [-0.063],
        dW1 = [[-0.00378, 0.00378], [0.00378, -0.00378]], db1 = [-0.00756, 0.00756]

Constraints

• Use the four formulas above, in order.
• Each gradient must match the shape of the parameter it updates.
• dW2 = outer(a1, delta2), dW1 = outer(x, delta1).

Notes

• Plug these into $\theta \leftarrow \theta - \eta\nabla_\theta L$ and you have one training step — the whole learning loop in a handful of lines.