Deep Neural Networks

Lesson 4

Loss functions

Measuring how wrong your model is, and why the choice matters

Loss functions, measuring how wrong the model is

You have layers, activations, and a forward pass that produces ŷ. The network is a machine, but who decides if the machine is right? This lesson follows a three-act story: why loss exists at all, why we start with mean squared error (MSE) for intuition, why that story breaks for confident probabilities, and why binary cross-entropy (BCE) is the usual hero for binary classification, designed to pair with a sigmoid output.

Act 1

Why do we even need a loss function?

We’ve built a net. It produces an output. Is it good?

Imagine showing a friend a number, 0.73, and asking: “Is this a good answer?” They cannot say yes or no until they know what was supposed to happen. A loss function is the ruler: it scores how far the model’s prediction sits from truth.

Loss = a single number (lower is better) that answers: “How wrong are we?”

Prediction ŷ	Actual y	Story
0.8	1	Small gap, feels “close enough” to start
0.2	1	Big gap, clearly off

Key idea (carry this forward)

Loss measures a kind of distance (not always Euclidean, depends on the formula) between prediction and truth. Training = adjust weights so that distance shrinks.

Mental model (preview):

Neural network ≈ function approximator
Loss ≈ objective (“what we want small”)
Training ≈ repeatedly minimize that objective

Act 2

Mean squared error, intuition first

We start here on purpose: geometry you can see, smooth curves you can optimize.

MSE: the friendly default for regression

For n examples, compare scalar y and ŷ:

L_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \bigl(y_i - \hat{y}_i\bigr)^2

Why it feels good:

Squaring makes big errors much more expensive than tiny ones (outliers shout loudly).
Symmetric: predicting too high vs too low by the same amount costs the same.
Smooth: as a function of ŷ (for fixed y), it is a parabola, derivatives exist and change gradually → friendly for gradient-based optimization later (backprop).

MSE loss vs prediction (fixed y = 1)

L(ŷ) = (ŷ − 1)², a smooth parabola with its minimum at ŷ = 1

⚠️ Deliberate discomfort: same label, different confidence

MSEBCEyour p̂

True label

Predicted prob p̂0.50

MSE = (p̂ − y)²

0.250

BCE = −[y·ln p̂ + (1−y)·ln(1−p̂)]

0.693

Suppose y = 1 (positive class). Compare ŷ = 0.9 vs ŷ = 0.6, both on the correct side of 0.5.

MSE says:

(1 - 0.9)^2 = 0.01, \qquad (1 - 0.6)^2 = 0.16

So MSE does penalize 0.6 more, but the ratio is modest compared to how humans feel: 0.9 is confident and right; 0.6 is hesitant. For probability outputs, we often want a loss that cares more about being sure while wrong than MSE’s pure squared gap story captures.

The tension

MSE is excellent for real-valued regression. For probabilities in classification, we want a loss that speaks the language of odds, logs, and calibration, enter cross-entropy.

Act 3

Binary cross-entropy, built for probabilities

BCE: “Don’t be confidently wrong”

For y ∈ 1 and ŷ ∈ (0, 1) (model probability of class 1):

L_{\text{BCE}} = -\frac{1}{n} \sum_{i=1}^{n} \Bigl[ y_i \log(\hat{y}_i) + (1 - y_i)\log(1 - \hat{y}_i) \Bigr]

(Implementations clip ŷ slightly inside (0, 1) so log stays finite.)

Intuition: log blows up near zero. If y = 1 but you predict ŷ = 0.01, then log(0.01) is hugely negative, after the minus sign, loss explodes. Confident wrong answers pay a steep price.

Situation (y = 1)	BCE story
ŷ = 0.99 (confident, correct)	Small loss, log(0.99) near 0
ŷ = 0.01 (confident, wrong)	HUGE loss, log(0.01) very negative

🔥 One-line teaching insight

BCE ≈ “Don’t be confidently wrong.”

Notice something interesting… (bridge to Lesson 3)

BCE expects ŷ to behave like a probability → must lie in (0, 1).
A sigmoid on the logit does exactly that.

So sigmoid + BCE is not a random pairing, it is a designed match between output semantics and loss semantics. (Same spirit as softmax + cross-entropy for multi-class.)

Regression

Linear output + MSE (or Huber, …)

Binary classification

Sigmoid probability + BCE

What the loss surface looks like

Two minima at (±1, 0) · saddle at (0, 0)

Step 0 / 90

L = (x²−1)² + ½y² ∇L = (4x(x²−1), y)lr = 0.07 · yellow path = gradient descent · ⬤ = current position

Deep network loss surfaces are non-convex. This double-well shows two equally valid minima and a saddle point at (0, 0). Starting near the saddle, gradient descent must choose a valley — a tiny perturbation in initialization can flip which minimum is reached. This is why random seed, learning rate, and batch noise all influence which solution a DNN finds, and why flat vs sharp minima matter for generalization.

🧩 How this sets up backprop

Forward pass → compute ŷ
Loss → compute how wrong
Backward pass → gradients of loss w.r.t. weights (chain rule through layers)

Next lesson territory: “We know how wrong we are, which way should each weight nudge?” That is gradient descent and backpropagation.

💡 Advanced layer (one line)

Binary cross-entropy corresponds to maximum likelihood under a Bernoulli model, minimizing BCE is equivalent to maximizing the log-probability of the labels given the predicted probabilities. Same bridge from neural nets → statistics → machine learning theory.

Check your understanding

1 / 9

Which loss function is the standard pairing for a network whose final layer is sigmoid and produces a single probability?

Practice it yourself

Loss is the ruler. Implement the scorers — and the numerically safe versions real frameworks ship — with hidden tests, no setup.

Pick the better modelEasy

Lower loss wins — the comparison training makes at every step.

Clipped binary cross-entropyEasy

Clip ŷ so log stays finite — the guard every BCE implementation uses.

BCE from logits (stable)Medium

Fuse sigmoid + BCE overflow-free — why you pass logits, not probabilities.