FreeUpdated 2 May 202648 minHard

Deep Neural Networks

Lesson 4

Loss functions

Measuring how wrong your model is, and why the choice matters

Loss functions, measuring how wrong the model is

You have layers, activations, and a forward pass that produces ŷ. The network is a machine, but who decides if the machine is right? This lesson follows a three-act story: why loss exists at all, why we start with mean squared error (MSE) for intuition, why that story breaks for confident probabilities, and why binary cross-entropy (BCE) is the usual hero for binary classification, designed to pair with a sigmoid output.

Act 1

Why do we even need a loss function?

We’ve built a net. It produces an output. Is it good?

Imagine showing a friend a number, 0.73, and asking: “Is this a good answer?” They cannot say yes or no until they know what was supposed to happen. A loss function is the ruler: it scores how far the model’s prediction sits from truth.

Loss = a single number (lower is better) that answers: “How wrong are we?”

Prediction ŷ	Actual y	Story
0.8	1	Small gap, feels “close enough” to start
0.2	1	Big gap, clearly off

Key idea (carry this forward)

Loss measures a kind of distance (not always Euclidean, depends on the formula) between prediction and truth. Training = adjust weights so that distance shrinks.

Mental model (preview):

Neural network ≈ function approximator
Loss ≈ objective (“what we want small”)
Training ≈ repeatedly minimize that objective

Act 2

Mean squared error, intuition first

We start here on purpose: geometry you can see, smooth curves you can optimize.

MSE: the friendly default for regression

For n examples, compare scalar y and ŷ:

L_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \bigl(y_i - \hat{y}_i\bigr)^2

Why it feels good:

Squaring makes big errors much more expensive than tiny ones (outliers shout loudly).
Symmetric: predicting too high vs too low by the same amount costs the same.
Smooth: as a function of ŷ (for fixed y), it is a parabola, derivatives exist and change gradually → friendly for gradient-based optimization later (backprop).

MSE loss vs prediction (fixed y = 1)

L(ŷ) = (ŷ − 1)², a smooth parabola with its minimum at ŷ = 1

⚠️ Deliberate discomfort: same label, different confidence

MSEBCEyour p̂

True label

Predicted prob p̂0.50

MSE = (p̂ − y)²

0.250

BCE = −[y·ln p̂ + (1−y)·ln(1−p̂)]

0.693

Suppose y = 1 (positive class). Compare ŷ = 0.9 vs ŷ = 0.6, both on the correct side of 0.5.

MSE says:

(1 - 0.9)^2 = 0.01, \qquad (1 - 0.6)^2 = 0.16

So MSE does penalize 0.6 more, but the ratio is modest compared to how humans feel: 0.9 is confident and right; 0.6 is hesitant. For probability outputs, we often want a loss that cares more about being sure while wrong than MSE’s pure squared gap story captures.

The tension

MSE is excellent for real-valued regression. For probabilities in classification, we want a loss that speaks the language of odds, logs, and calibration, enter cross-entropy.

Act 3

Binary cross-entropy, built for probabilities

BCE: “Don’t be confidently wrong”

For y ∈ 1 and ŷ ∈ (0, 1) (model probability of class 1):

L_{\text{BCE}} = -\frac{1}{n} \sum_{i=1}^{n} \Bigl[ y_i \log(\hat{y}_i) + (1 - y_i)\log(1 - \hat{y}_i) \Bigr]

(Implementations clip ŷ slightly inside (0, 1) so log stays finite.)

Intuition: log blows up near zero. If y = 1 but you predict ŷ = 0.01, then log(0.01) is hugely negative, after the minus sign, loss explodes. Confident wrong answers pay a steep price.

Situation (y = 1)	BCE story
ŷ = 0.99 (confident, correct)	Small loss, log(0.99) near 0
ŷ = 0.01 (confident, wrong)	HUGE loss, log(0.01) very negative

🔥 One-line teaching insight

BCE ≈ “Don’t be confidently wrong.”

Notice something interesting… (bridge to Lesson 3)

BCE expects ŷ to behave like a probability → must lie in (0, 1).
A sigmoid on the logit does exactly that.

So sigmoid + BCE is not a random pairing, it is a designed match between output semantics and loss semantics. Softmax + cross-entropy is the same match for multi-class, worked through in full just below.

Regression

Linear output + MSE (or Huber, …)

Binary classification

Sigmoid probability + BCE

Multi-class

Softmax distribution + cross-entropy

Multi-class: softmax + cross-entropy

Softmax turns $K$ logits into a distribution:

p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} \qquad\qquad p_k > 0, \quad \sum_{k=1}^{K} p_k = 1

Cross-entropy scores it against a one-hot label. Every term with $y_k = 0$ vanishes:

L \;=\; -\sum_{k=1}^{K} y_k \log p_k \;=\; -\log p_c

Substitute the softmax and split the log:

L \;=\; -\log \frac{e^{z_c}}{\sum_j e^{z_j}} \;=\; \log \sum_{j=1}^{K} e^{z_j} \;-\; z_c

Only $p_c$ enters. The other classes act through the denominator alone.

Worked example

\mathbf{z} = [\,2.0,\;\; 1.0,\;\; 0.1\,], \qquad c = 0

	k = 0	k = 1	k = 2
z	2.0	1.0	0.1
e^z	7.389056	2.718282	1.105171
p = e^z / 11.212509	0.659001	0.242433	0.098566
L if c = k	0.4170	1.4170	2.3170

L \;=\; -\log 0.659001 \;=\; 0.4170 \qquad\qquad \log 11.212509 - 2.0 \;=\; 0.4170

The last row is spaced by the logit gaps, $1.0$ and $0.9$ : the $\log\sum_j e^{z_j}$ term is shared, only $z_c$ moves.

Numerical stability

$e^{1000}$ overflows. Shift by $m = \max_j z_j$ , which cancels top and bottom:

p_k = \frac{e^{z_k - m}}{\sum_j e^{z_j - m}} \qquad\qquad L = \log \sum_{j} e^{z_j - m} \;-\; (z_c - m)

m = 2.0 \;\Rightarrow\; e^{\mathbf{z}-m} = [\,1,\;\; 0.367879,\;\; 0.149569\,], \quad \textstyle\sum = 1.517448, \quad p \text{ unchanged}

The largest exponent is now $e^0 = 1$ .

Why the loss takes logits

CrossEntropyLoss applies softmax internally to use this shift. Feed it softmax output and softmax runs twice, silently, with no error raised.

Gradient

\frac{\partial L}{\partial z_k} = p_k - y_k

[\,0.659,\;\; 0.242,\;\; 0.099\,] \;-\; [\,1,\;\; 0,\;\; 0\,] \;=\; [\,-0.341,\;\; 0.242,\;\; 0.099\,]

Same form as sigmoid + BCE, which is the $K = 2$ case:

p_1 = \frac{e^{z_1}}{e^{z_0} + e^{z_1}} = \frac{1}{1 + e^{-(z_1 - z_0)}} = \sigma(z_1 - z_0)

Python

First run loads the Python runtime (~10 MB) — takes ~5–10 seconds. Subsequent runs are instant.

Set c = 2 and the loss goes $0.4170 \to 2.3170$ on the same forward pass.

What the loss surface looks like

Two minima at (±1, 0) · saddle at (0, 0)

Step 0 / 90

L = (x²−1)² + ½y² ∇L = (4x(x²−1), y)lr = 0.07 · yellow path = gradient descent · ⬤ = current position

Deep network loss surfaces are non-convex. This double-well shows two equally valid minima and a saddle point at (0, 0). Starting near the saddle, gradient descent must choose a valley — a tiny perturbation in initialization can flip which minimum is reached. This is why random seed, learning rate, and batch noise all influence which solution a DNN finds, and why flat vs sharp minima matter for generalization.

🧩 How this sets up backprop

Forward pass → compute ŷ
Loss → compute how wrong
Backward pass → gradients of loss w.r.t. weights (chain rule through layers)

Next lesson territory: “We know how wrong we are, which way should each weight nudge?” That is gradient descent and backpropagation.

💡 Advanced layer (one line)

Binary cross-entropy corresponds to maximum likelihood under a Bernoulli model, minimizing BCE is equivalent to maximizing the log-probability of the labels given the predicted probabilities. Same bridge from neural nets → statistics → machine learning theory.

Check your understanding

1 / 14

Which loss function is the standard pairing for a network whose final layer is sigmoid and produces a single probability?

Practice it yourself

Loss is the ruler. Implement the scorers — and the numerically safe versions real frameworks ship — with hidden tests, no setup.

Pick the better modelEasy

Lower loss wins — the comparison training makes at every step.

Clipped binary cross-entropyEasy

Clip ŷ so log stays finite — the guard every BCE implementation uses.

BCE from logits (stable)Medium

Fuse sigmoid + BCE overflow-free — why you pass logits, not probabilities.