Rate of change and the derivative

In The only calculus you need you dragged a ball around a bowl and read its tilt. That lesson gave you the picture, and handed you one formula already written — 2 * (w - 3) — to type into a box. It never showed you where that formula came from, and it never showed you what to do when nobody hands you one.

So take the picture away. Below is the same bowl with almost all of it hidden. You can see one point and a thumbnail of curve either side of it, and nothing else.

One of those two questions has an answer and the other does not. The patch tells you which way the ground falls away. It does not tell you where the bottom is, or how far off it is, and no amount of squinting at it will.

So here is the question this lesson answers. That local tilt is clearly worth a number — how do you actually get the number, from the function alone?

Difference quotient: a slope from one nudge

Work with a concrete function. This is the same bowl as before:

Read it as a loss L that depends on one knob w. Start at w = 0, where L(0) = (0 − 3)² = 9.

Now nudge. Move w by a small amount and see what L does. Write Δw (delta w) for the change in the input and ΔL for the change in the output; Δ is a capital Greek delta and it just means the change in. Take a nudge of Δw = 0.1:

The loss went down by 0.59 when the knob went up by 0.1. Those two numbers mean nothing separately. Divide them and they mean something:

That is rise over run — the slope of the straight line through the two points you just visited. It has a name, the difference quotient, and it is the whole computation. Shrink the nudge to Δw = 0.01 and L(0.01) = 8.9401, so ΔL = −0.0599 and the quotient is −5.99.

The two answers are drifting toward something. You can see exactly what, because the algebra here is short. Call the nudge h, so Δw = h, and expand with u = w − 3:

The estimate is the number 2(w − 3), plus your nudge. The error is exactly the size of the nudge, so the smaller the nudge the closer you are. At w = 0 that target number is 2(0 − 3) = −6, which is why 0.1 gave −5.9 and 0.01 gave −5.99.

Shrinking h toward zero without ever setting it to zero is called taking a limit, and the limit is what turns this estimate into an exact statement. That is all this course will say about limits.

Reading the slope at w = 0, 3 and 6

You now have a formula that works at every point, not just at zero. Write it as L′(w), read aloud as L prime of w:

That is the formula you were handed in The only calculus you need, and now you know where it came from. Evaluate it at three places on the bowl:

Look at the outer two rows. Both sit at a loss of 9. The loss value is identical and tells you nothing about where to go; the slope is opposite in sign and tells you everything. The middle row also confirms the formula: 2(w − 3) is zero exactly when w = 3.

Derivative: the slope of the tangent line

Here is what that number is, in the picture.

Pick your point on the curve and pick a second point a nudge h away. The straight line through those two points is a secant, and its slope is the difference quotient you just computed. Now slide the second point in toward the first. The line pivots, and it settles onto the one straight line that touches the curve at your point and matches its direction there. That line is the tangent, and its slope is the number the difference quotients were converging to.

Each rung of the ladder is closer than the last, and the gap is exactly the size of the nudge. The line stops pivoting when there is no nudge left to shrink.

That number is the derivative of L at w. Both statements below say the same thing:

The word local is doing real work. A tangent line is fitted at one point using only the curve immediately around that point, so the derivative is a fact about a neighbourhood and not about the function as a whole. That is why the hidden-patch view at the top could answer which way and could not answer how far to the bottom. A slope is a direction, not a destination.

Sign and size of the derivative: direction and step

One number, and an optimizer reads two separate things off it.

Its sign is the direction. A positive slope means the function climbs as the input grows, so downhill is toward smaller inputs. A negative slope means the reverse. That is the whole of the rule, and it explains the minus sign in the update that gradient descent runs:

The arrow means replace w by, and η is the Greek letter eta, the learning rate, a small positive constant. Subtracting a negative slope moves w up; subtracting a positive slope moves w down. Either way you go downhill without anyone telling you where down is.

Its size is the step. η is fixed, so the distance moved is η times the size of the slope, and a steeper point moves further. With η = 0.1:

From w = 0 the step is 0.6. From w = 2, where the slope is three times gentler, the step is 0.2. The steep wall is crossed quickly and the approach to the bottom slows down on its own, because the slope shrinks as you get closer. Nothing schedules that. The sign picks the direction and the size picks the distance, and those two readings are the entire contribution calculus makes to training.

Notation: f′(x) and df/dx

You will meet the same derivative written two ways, and the difference is spelling rather than substance.

The first is Lagrange's, and it is compact: a prime mark on the function name. The second is Leibniz's, and it names the input variable out loud — the derivative of f with respect to x. For this lesson's bowl the two read L′(w) and dL/dw, and both equal 2(w − 3).

Two spellings survive because they are good at different jobs. When a function has one input, the prime is shorter and nothing is ambiguous. When a function has several inputs, or when the chain rule is stacking one function on another, you need to say which variable you are moving, and dL/dw says it in the symbol. The Leibniz form also keeps the shape of where it came from: dL/dw is the ratio ΔL/Δw after the nudge has been taken to its limit. It is written like a fraction because it is the ghost of one — but dL and dw are not two numbers you computed and divided, so it is one symbol, not a division you can pull apart.

Finite difference: why shrinking h stops helping

The algebra said the error is exactly h, so a smaller nudge is always a better nudge. A computer disagrees. Here is the same estimate of L′(0) at five nudges, where the true answer is −6:

The first three behave exactly as promised. Then it turns around. At h = 1e−12 the estimate is over five hundred times worse than at h = 1e−6, and at h = 1e−15 the first decimal place is wrong.

The cause is the subtraction. A floating point number carries about sixteen significant digits. At h = 1e−12, L(0 + h) and L(0) are both about 9 and differ in the twelfth decimal place, so subtracting them cancels away almost every accurate digit and leaves a handful of survivors carrying all the rounding noise. Dividing that wreckage by 1e−12 multiplies the noise by a trillion. Two errors are fighting: the approximation error, which shrinks with h, and the rounding error, which grows as h shrinks. The best h is where they cross, and it is nowhere near zero.

So the nudge is a fine way to understand a derivative and a poor way to compute one. What you want is the exact expression, 2(w − 3), obtained from the function without ever touching a number. Getting that for functions you have not personally expanded by hand is what derivative rules are for, and that is the next lesson in this course.

In code

The estimate, the true value, and the turn. Change w to 6 and predict both the sign and the size before you run it.

At w = 0.0 the three lines print estimates of -5.990000000000, -5.999999000750 and -6.000533403494, with errors 1.00e-02, 9.99e-07 and 5.33e-04. Read the error column downward: it falls by a factor of ten thousand, then rises again. The middle row is the best this method can do here, and the last row is the trap.

slope_estimate never looks inside L. It calls it twice and divides, which is why it works on any function at all — and why it can only ever approximate. Later, in How neural networks learn (backward pass), you will meet this exact quantity computed for every weight in a network at once, and it is computed from rules rather than from nudges, for precisely the reason the last row of that table shows.