Backprop — the output delta

Background

With a sigmoid output and MSE loss, the first backprop quantity is the output delta — the gradient of the loss w.r.t. the output pre-activation $\mathbf{z}_2$:

$$\boldsymbol{\delta}_2 = \underbrace{(\mathbf{a}_2 - \mathbf{y})}_{\partial L/\partial \mathbf{a}_2}\;\odot\;\underbrace{\mathbf{a}_2(1 - \mathbf{a}_2)}_{\sigma'(\mathbf{z}_2)}$$

The first factor is "how wrong, and in which direction"; the second is the local sigmoid slope $\sigma'(z_2) = a_2(1-a_2)$. Their elementwise product is the error signal that flows back into the output layer.

Problem statement

Implement output_delta(a2, y) returning $\boldsymbol{\delta}_2 = (\mathbf{a}_2 - \mathbf{y})\odot \mathbf{a}_2(1-\mathbf{a}_2)$.

Input

• a2 — array-like prediction (sigmoid output), each in $(0,1)$.
• y — array-like true labels, same length.

Output

Returns an np.ndarray (same length): the output delta.

Examples

Example 1

Input:  a2 = [0.8], y = [1]
Output: [-0.032]

Explanation: $(0.8-1)\cdot0.8\cdot0.2 = -0.2\cdot0.16 = -0.032$.

Example 2

Input:  a2 = [0.5, 0.5], y = [1, 0]
Output: [-0.125  0.125]

Explanation: error $\mp0.5$ times slope $0.25$.

Constraints

• Compute (a2 - y) * a2 * (1 - a2) elementwise.
• Return an np.ndarray.

Notes

• A perfectly correct, saturated prediction gives $\delta_2\approx0$ on both factors — no error and no slope — so it stops learning, which is exactly right.