Foundations of Regression

Lesson 2

Why do we need logistic regression?

Where linear regression fails for binary outcomes

Why do we need logistic regression?

Linear regression is great for continuous target variables (e.g., predicting house prices), but it fails when dealing with classification problems (e.g., predicting if an email is spam or not).

Linear regression fits

Continuous outcomes, price, temperature, demand, where any real-valued prediction can make sense (possibly after clipping in post-processing).

Classification needs

Discrete labels, spam / not spam, and usually probabilities over classes, not unbounded scores.

Problems with linear regression for classification

Output is unbounded

The mismatch

A linear model ŷ = wᵀx + b can output any real number, 5, −0.3, even π, but a class probability must live in [0, 1]. That single mismatch is why linear regression can't classify.

Solution: We need a mechanism to map those real numbers into probabilities, for example, an output that always says the probability of a data point belonging to class 0 is 0.4 and the probability of belonging to class 1 is 0.6.

Class 0

P = 0.4

Class 1

P = 0.6

Two-class probabilities should live in [0, 1] and (for binary problems) sum to 1.

So we need a function that takes any real score, however large or negative, and squishes it into the [0, 1] range a probability lives in. Drag the raw score below and watch what comes out, it never escapes 0 or 1:

00.51z → −∞z → +∞raw score z

Drag the raw score. Notice it never escapes 0 or 1.

probability = 0.50

leans class 1 · 0.5 is the toss-up

A score of any size goes in; a value in (0, 1) always comes out. That squishing curve is the sigmoid.

Non-linearity

First we need a way to represent non-linear structure in the world, and plain linear models like linear regression can't really capture that on their own. Second, we often want decision boundaries that separate regions of the input space by class, and treating classification as raw linear regression on labels doesn't give us a clean probabilistic separator.

Most real-world data is not cleanly split by a straight line. Toggle between the two datasets below. A straight boundary handles the separable one perfectly, but on tangled data no straight line beats a coin flip, you need a curve:

Straight-line accuracy: 100%

Rotate until the line splits the two clusters. One straight cut reaches 100%. This is the world a linear model can fit.

NeedPlain linear regression on labels
Curved or tangled class structureA single linear score wᵀx + b may be the wrong shape unless we add features or use another model.
Clear "which side am I on?" ruleWe want a boundary tied to probabilities, not arbitrary real line targets like 0 and 1.

From linear score to probability: the sigmoid bridge

To fix these problems, we squash the output of a linear model into a probability using the sigmoid function, which leads us to logistic regression.

Let z = wᵀx + b be the same linear score you would use in linear regression (for one sample). The sigmoid is:

σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}

Key facts (easy to check from the formula):

σ(z)∈(0, 1),lim⁡z→−∞σ(z)=0,lim⁡z→+∞σ(z)=1.\sigma(z) \in (0,\,1), \qquad \lim_{z \to -\infty} \sigma(z) = 0, \qquad \lim_{z \to +\infty} \sigma(z) = 1.

For binary classification, a standard choice is to interpret:

P(y=1∣x)=σ(w⊤x+b).\mathbb{P}(y = 1 \mid \mathbf{x}) = \sigma\bigl(\mathbf{w}^\top \mathbf{x} + b\bigr).

Then (for two classes):

P(y=0∣x)=1−σ(w⊤x+b).\mathbb{P}(y = 0 \mid \mathbf{x}) = 1 - \sigma\bigl(\mathbf{w}^\top \mathbf{x} + b\bigr).

That smooth S-curve is exactly the squisher you dragged earlier: small z maps near 0, large z near 1, and the output is always a bounded probability.

The set of points where σ(wᵀx + b) = 0.5 (equivalently wᵀx + b = 0) is still a linear decision boundary in x, logistic regression is linear in the features but non-linear in the probability because of σ. Richer boundaries usually mean richer features (e.g. polynomials) or a different model, but we still start from this clean score → probability idea.

Check your understanding

1 / 8

What is the main problem with using linear regression for binary classification?

Practice it yourself

You saw why a raw linear score can't be a probability. Now pin it down in code with two Easy drills, with real Python and hidden tests, no setup.

When a linear score isn't a probabilityEasy

Flag which raw scores escape [0, 1] — the failure that motivates the sigmoid.

Two-class probabilitiesEasy

Given P(y=1), return both class probabilities — they always sum to 1.

Test your understanding

Prof is ready

Prof will ask you questions about Why do we need logistic regression? — not explain it. You'll be surprised what you don't know until you have to say it.

Finished this lesson?

Read through the lesson first (0/20s).