Derivative rules for ML

Twice already, PROF has handed you a derivative and kept walking. In How neural networks make predictions (forward pass):

the derivative we need everywhere is the same tidy expression, σ'(z) = σ(z)(1 − σ(z)) = a(1 − a)

And in Linear regression: lines, SSR, and gradient descent, a squared residual is differentiated in front of you and the result explained in a single line:

the 2 × residual comes from differentiating the square (the derivative of x² is 2x)

Both are correct. Both hand you a result instead of a method. The second at least echoes something you have done — in Rate of change and the derivative you expanded a square by hand and watched a factor of 2 fall out. The first you have never seen built at all.

So here is the question this lesson answers. Which handful of rules covers the functions a model is made of, and how far do those rules actually get you?

Five rules: c, ax, x², eˣ and log x

One more piece of notation first. Writing d/dx in front of an expression means the derivative of this, with respect to x. It is the same object as f′(x) and df/dx, just written in front of the thing rather than after it, which is convenient when the thing has no name.

A rule is an exact statement about a whole function, not an estimate at one point. Each of the five below has a picture behind it, and the picture is the part worth keeping.

A constant does not change. Its graph is a flat line, so nudging x moves the output by nothing at all, and a change of zero divided by any nudge is zero.

A line has the same steepness everywhere. Nudge x by h and ax becomes a(x + h), a change of ah; divide by h and the h cancels exactly, at any size, so no shrinking is required.

x² steepens as you move away from zero, and 2x is that steepening measured. You have already done this expansion: (x + h)² − x² = 2xh + h², which over h is 2x + h, and the leftover h vanishes as the nudge shrinks. At x = 1 the slope is 2; at x = 3 it is 6. Three times further out, three times steeper.

eˣ is the curve whose slope equals its own height. Here e is the fixed number 2.71828…, and this property is exactly what singles it out from every other base: at x = 0 the height is 1 and so is the slope; where the curve is at 20, it is climbing at 20.

log x flattens as x grows, and 1/x is that flattening. Throughout this course log means the natural logarithm, base e — the inverse of eˣ, defined for x > 0. At x = 1 its slope is 1, at x = 10 it is 0.1, at x = 100 it is 0.01. The curve keeps rising forever and keeps rising more slowly forever.

Five functions, five slopes. Read each pair together — the function on the left, its slope on the right — and watch what the slope is doing while the function moves.

The two flat cases are the ones to hold on to. For a constant the slope readout never moves off zero, and for a line it never moves off a. Everything else in this lesson is those five rules being combined.

Sum rule: derivatives split across added terms

Add two functions and their slopes add too.

The reason is the nudge. Move x by h and f changes by some amount, g changes by some amount, and the sum changes by the two amounts added. Divide that by h and the fraction splits into two quotients, one per term. A constant factor rides along the same way: scale every output by c and every change is scaled by c, so d/dx (c·f) = c·f′.

That makes compound expressions routine. For g(x) = x² + 3x − 4, differentiate term by term: 2x, then 3, then 0, so g′(x) = 2x + 3 and g′(2) = 7.

Now the part that matters for training. A loss over a dataset is a sum — one term per example. The sum rule says its derivative is the sum of the per-example derivatives, so the gradient of a dataset loss is the sum of the gradients of its examples, each computable on its own and added in afterwards. Nothing forces you to hold all of them at once, which is what makes mini-batch training legitimate rather than a convenient approximation.

Differentiating a squared residual: −2(y − ŷ)

Two symbols before the arithmetic. Write y for the true value an example carries, and ŷ, read aloud as y-hat, for what the model predicted. Their difference y − ŷ is the residual, and squaring it gives the squared error (y − ŷ)² that regression sums up.

Differentiate it with respect to ŷ, since ŷ is the part the model controls. Nudge ŷ by h and write e for the residual y − ŷ, so that y − (ŷ + h) is e − h:

The same expansion as x², with one sign flipped by the subtraction. Shrink the nudge and the leftover h goes:

Sit with what that says. Take a true value y = 5 and evaluate at three predictions:

The derivative column is the residual column, doubled and sign-flipped. Nothing else survives the differentiation. The gradient of squared error is the error itself, which is why training code computes a residual and then uses very nearly that same array as a gradient. It is also the missing half of the quote at the top: the 2 × residual is this, and the −xᵢ sitting beside it in that lesson comes from the chain rule, which is where this course goes next.

Deriving σ′(z) = σ(z)(1 − σ(z))

The sigmoid is written σ(z), with σ the Greek letter sigma and z the input:

Three small pieces build its derivative, and you have two of them already.

First, d/dz e^−z = −e^−z. The graph of e^−z is eˣ with the input axis reversed, so nudging z upward moves −z downward by the same amount, and every change comes out with its sign flipped.

Second, name the denominator u = 1 + e^−z. By the sum rule the constant 1 contributes 0, so du/dz = −e^−z.

Third, the reciprocal 1/u, which is not one of the five. Derive it once, by nudging u:

Now σ is 1/u, and u itself depends on z. When one function feeds another, the two slopes multiply — that is the chain rule, which this course states in general in The chain rule; here we need only this one instance of it:

Correct, and unrecognisable. Tidy it using σ itself. Since σ = 1/u, the quantity 1 − σ is (u − 1)/u, and u − 1 is exactly e^−z, so 1 − σ = e^−z/u. Multiply the two:

Same expression. So the derivative, written σ′(z) and read sigma prime of z:

That is the line the forward-pass lesson asserts, now built from the rules above. σ′ is computed from σ, not from z — a layer that has already produced its output has already produced its own slope, and backpropagation gets it for one multiplication.

Why σ′(z) never exceeds 0.25

Because 1 + e^−z is always greater than 1, σ(z) always lands strictly between 0 and 1. Write s for σ(z) and the derivative is s(1 − s), a product of two numbers that add to 1. That product has a hard ceiling, and one line of algebra shows it:

A square is never negative, so the subtraction can only take something away. σ′ never exceeds 0.25, and reaches it only when s = 1/2, which happens at z = 0 and nowhere else. Away from the middle it collapses fast:

Hold on to 0.25, because it comes back. Later, in How neural networks learn (backward pass), the gradient travelling backwards through a deep network picks up one factor of σ′ per layer. Ten sigmoid layers means a product of ten numbers each no larger than a quarter — at best 0.25¹⁰, which is about 9.5 × 10⁻⁷. That is the arithmetic behind vanishing gradients, and it is a fact about this one derivative.

In code

Two ways to get the same number: the rule, and the nudge from Rate of change and the derivative. Change the z values and predict the rule column before you run it.

The two columns agree to all six decimals — 0.017663, 0.104994, 0.250000, 0.104994, 0.017663 — and the scan reports a largest value of 0.250000, matching the algebra rather than merely being consistent with it. The last line prints 9.537e-07. The difference between the columns is not accuracy but cost: by_nudge calls sigma twice and knows nothing about it, while by_rule reuses a value the network has already computed.

Five rules, a rule for splitting sums, and one reciprocal derived on the spot got you both derivatives that PROF had been asserting. What they cannot do is handle a function fed into another function — σ(wx + b) rather than σ(z) — which is every layer of every network. That composition is the chain rule, and it is what this course builds next.