The chain rule

One neuron, with every number filled in. The input is x = 2, the target is y = 1, the weight is w = 0.3 and the bias is b = −0.1. Three things happen, in order:

Run it left to right. z = 0.6 − 0.1 = 0.5, then a = σ(0.5) = 0.6225, then L = (1 − 0.6225)² = 0.1425.

Now nudge the weight. Take w from 0.30 to 0.31 and run all three stages again. The loss lands at 0.1390, lower by 0.0035. Divide the change by the nudge, exactly as you did in Rate of change and the derivative, and you get about −0.352. Push this weight up, the loss goes down.

That single number is what training runs on, and you got it by running the network twice. What you did not get is any account of where it came from. Three stages sit between w and L, and each one did something to the nudge as it passed: the line stretched it, the sigmoid squashed it, the square scaled it. Run-it-twice reports the total and hides the parts — and a network with a million weights cannot afford to be run a million times.

So here is the question this lesson answers. When a change travels through a chain of functions, how do the stages combine?

Function composition: f(g(x))

Start by naming the object, because the object is what the rule is about.

When the output of one function becomes the input of another, the result is a composition. Written out, f(g(x)) means: feed x to g, take whatever comes out, feed that to f. The shorthand is (f ∘ g)(x), and the small circle is read after — f after g. Order matters, and it runs right to left, the same way the matrix product AB means apply B first in Matrix multiplication and the transpose.

The three stages at the top of this lesson are one composition wearing three names. Substitute each into the next and the intermediate values disappear:

Read that from the inside out. The innermost bracket is the pre-activation wx + b, the sigmoid wraps that, the subtraction from y wraps the sigmoid, and the square wraps everything. Four nested boxes, and w sits in the innermost one.

A forward pass is a composition, from the first weight to the final loss. That is not an analogy for one; it is literally the same object, and it is why one rule covers a linear model, a deep network and an RNN alike.

Slopes multiply along a chain: the gear ratio

Before any calculus, a machine with no numbers hidden in it.

Two meshed gears. The first turns three times for every one turn of the input shaft. The second turns twice for every one turn of the first. Turn the input once and the output turns — how many times?

Six. One input turn drives three turns of the first gear, and each of those drives two turns of the second, so you count 3 × 2 and not 3 + 2. Adding is the intuition almost everyone reaches for, and it is wrong for a specific reason: the second gear does not receive the input's motion, it receives the first gear's motion, already tripled. Each stage acts on what reached it, so each stage scales what reached it.

Add a third gear at 1.5 and the end-to-end ratio is 9. Change that third one from 1.5 to 0.5 and the ratio falls from 9 to 3. Along a chain, local ratios multiply — so a single stage below 1 drags the whole train down, however fast the others spin.

Now the case worth remembering. Set any ratio to exactly 0 and the output stops dead, no matter how fast every other gear spins, because zero times anything is zero. That is not a curiosity. A ReLU unit whose input is negative has a slope of exactly 0, and everything behind it in the chain learns nothing at all.

Adding does happen in calculus, but not here. It happens across separate terms that are summed — that is the sum rule from Derivative rules for ML — and never along a chain, where one function feeds the next.

The chain rule: (f ∘ g)′(x) = f′(g(x)) · g′(x)

The gear train, written as calculus, is the whole rule:

Two factors, one per stage. g′(x) is the inner function's slope at the point you are standing on: nudge x, and this is how much g moves. f′(g(x)) is the outer function's slope — but read where it is measured. Not at x. At g(x), the value the inner function actually handed over, because that is the input f is sitting at. The outer factor is evaluated at the inner value, and getting that wrong is the single most common error in applying the rule.

Take f(u) = u² and g(x) = 3x + 1, so f(g(x)) = (3x + 1)². The outer slope is f′(u) = 2u, evaluated at u = 3x + 1. The inner slope is g′(x) = 3, from the rule for a line.

The 2 came from the square, and the 3 came from the inside. At x = 2 the value is 6 × 7 = 42.

Check it without the rule. Expand first: (3x + 1)² = 9x² + 6x + 1, then differentiate term by term with the sum rule to get 18x + 6, and at x = 2 that is 42 again. The chain rule is not a new fact about (3x + 1)² — it is a way to reach the same answer without ever expanding, which matters because σ(wx + b) cannot be expanded into a sum of terms the five rules cover.

Three local slopes: ∂L/∂a · ∂a/∂z · ∂z/∂w

Three stages instead of two costs nothing new: apply the rule twice and you get three factors. In general, a chain of any length contributes one factor per link.

Here it is for the neuron at the top of this lesson, in the notation PROF's deep learning course uses:

One symbol first. Read ∂ as partial. It is the same derivative you already have — nudge one input, see how the output moves — carrying one extra piece of bookkeeping: L depends on w and on b, and ∂L/∂w says that b was held still while w moved. The ∂ changes nothing about the arithmetic; it records which knob was turned. Later in this course, a lesson gathers all of a function's partial derivatives into one object. Here the symbol is only a label.

Now read the equation as a sentence, right to left, following the nudge. Move w a little: z moves ∂z/∂w times as much. That movement in z moves a by ∂a/∂z times as much again. And that movement in a moves L by ∂L/∂a times as much. Three ratios, three gears, one product.

Every factor is a derivative rule you already have:

Multiply the three:

Compare that with the −0.352 you got at the top by running the network twice. The gap is the nudge. That estimate used h = 0.01 and sits 0.003 away from the true slope; halve the nudge and the gap halves too — at h = 0.001 it is 0.0003 — which is exactly the behaviour Rate of change and the derivative predicted. The rule gives the number the nudge was converging to, from one forward pass and three multiplications, with no second run of the network.

Look at the middle factor. It is 0.235, and it can never be more than 0.25 — that is the sigmoid ceiling proved in Derivative rules for ML. Whatever the loss and the input contribute, this stage can only ever shrink what passes through it.

Long chains: 0.25²⁰, 0.9²⁰ and 1.1²⁰

Stack layers and you stack factors. A network twenty layers deep puts twenty links between an early weight and the loss, and the slope reaching that weight is the product of all twenty local slopes.

So multiply one number by itself and watch. Here is the end-to-end factor for a chain where every link contributes the same amount:

Three different fates, from three numbers that all look close together. At 0.25 — the best a sigmoid layer can ever do — twenty links leave roughly one part in a trillion of the signal. At 0.9, a mild shrink per link, twenty links still cost 88% of it. At 1.1, a mild growth per link, the signal is nearly seven times larger and climbing. Only 1.0 holds still, and nothing in a real network sits at exactly 1.0.

Depth does not add up; it compounds. That collapse has a name — a vanishing gradient — and it is not a bug in PyTorch, a rounding error, or a flaw in anyone's implementation. It is many numbers below one, multiplied, and the arithmetic would come out the same way on paper. The fix has to change the numbers, which is why ReLU (whose slope is 1 wherever it is active) and LSTM gates exist at all, and why those are architectural decisions rather than debugging.

In code

The three factors, assembled by hand, and checked against a nudge. Then set w to -1.5, where the prediction is much further from the target, and predict what that does to dL/dw before you run it.

The product prints -0.3548938347 and the nudge prints -0.3548938347, agreeing to ten decimal places with a gap of 2.40e-11. Two independent routes to one number, which is the check worth keeping: whenever a hand-derived slope disagrees with a nudge, the derivation is wrong.

Now run it at w = -1.5. The prediction drops to 0.043, the residual grows from 0.378 to 0.957, and dL/da more than doubles to -1.9137854901 — yet dL/dw falls to -0.1578835500, because da/dz has collapsed from 0.235 to 0.041. A worse prediction produced a weaker slope. That is saturation, and it is one factor of the product overruling another.

The two routes are not equally cheap. nudged runs the whole forward pass twice and would need two more runs for b, and two more for every other parameter. backward runs the forward pass once and reads all three factors off values that pass already produced — a and x were sitting there. That saving, applied to every weight in a network at once, is the reason training is affordable.

So go back to the pipe. The loss moved by −0.355 per unit of w, and now the movement has an itemised bill: −0.755 from the squared error, 0.235 from the sigmoid, 2 from the line. The chain rule is a rule of calculus, and backpropagation is the algorithm that applies it along a network's chain of intermediate values — which is very nearly word for word how How neural networks learn (backward pass) describes itself, and you can now read that sentence as a statement rather than a slogan. The same rule reappears in an RNN, where the chain runs backwards through time instead of through layers, and nothing about it changes except its length.

One thing here was quietly incomplete. This neuron has two knobs, w and b, and we froze b to look at w. Freezing everything else and turning one knob at a time is what ∂ was recording, and doing it properly — for every parameter, all at once — is the next lesson in this course.