Linear regression

Linear regression, lines, SSR, and gradient descent

Understanding how a line is represented in linear regression

Let's start with a simple example:

Here, we're not trying to classify anything, we're predicting a continuous value. This is regression, not classification.

The equation of a line (1D input)

When there is only one input feature, the relationship is modeled as:

Or more generally, in machine learning terms:

Moving beyond 1D: multiple input features

What happens when we have more than one input?

• X₁ = size (sq. yards)
• X₂ = location score
• X₃ = city weight

The equation becomes:

In compact vector form (weights w ∈ ℝⁿ, input x ∈ ℝⁿ):

This is still a linear model, but it's now a hyperplane in higher dimensions:

Each wᵢ is a parameter that the model learns during training.

Visualizing a line

Consider the equation:

This is a classic slope-intercept form of a line:

It represents a straight line in 2D space, and helps build intuition before moving into higher-dimensional regression.

Sum of squared residuals (SSR) in linear regression and how it helps find optimal coefficients

1. Understanding residuals

In linear regression, our goal is to find the best-fitting line:

where:

• y is the predicted output.
• x is the input.
• m (slope) and c (intercept) are the coefficients we want to optimize.

In general, this equation represents the equation of a line in 2D space where:

• m is the slope (how much y changes when x increases).
• c is the y-intercept (the value of y when x = 0).

This equation defines a straight line because:

Linear regression: an informal explanation

Linear regression is a technique that helps us find the best straight line through a set of data points. Let me explain how it works in a more structured and clear way.

What are we trying to do?

We want to find a line with the equation Y = mx + c (where m is the slope and c is the y-intercept) that best fits our data.

Our process

Our dataset

Training data:

Testing data (with current prediction errors):

The improvement process

We start with m = 1 and c = 1, which gives us poor predictions. For example, with these values, when x = 51, we get y = 52 (which is far from the true value of 149).

So we begin the iterative process:

• Calculate how wrong our predictions are
• Adjust m slightly
• Adjust c slightly
• Test if our predictions improved
• Repeat

Each iteration makes our line fit the data better. After multiple iterations, our m and c values will converge to the optimal values that minimize the errors across all data points.

The pattern in our data suggests m should be close to 3 and c should be around 0, which would give us Y ≈ 3x. This matches what we can observe: when x doubles, y approximately triples (e.g., x = 100 → y = 296, x = 200 → y = 596).

This step-by-step refinement is the essence of the gradient descent algorithm commonly used in machine learning.

Residuals and SSR

For each data point (xᵢ, yᵢ), the residual is the difference between the actual value yᵢ and the predicted value:

With prediction ŷᵢ = m xᵢ + c (same line as y = mx + c):

Since some residuals may be positive and others negative, we square them to measure the total error. Sum of squared errors (SSR):

This is also called the cost function or loss function:

Drag m and c below. Each red line is a residual (true − predicted), and each red square is that residual squared, so its area is the squared error. The SSR is the total area of all the squares. Try to shrink it as much as you can, and notice how a single far-off point (a big square) dominates the total.

2. Finding the optimal coefficients using gradient descent

The best values for m and c are the ones that minimize SSR. We use gradient descent to iteratively adjust m and c until we reach the optimal values.

Gradient descent steps

Where do these gradient formulas come from?

Step 3 plugged in two derivatives. Here is where they come from. Start from the cost, one squared residual per point:

Nudging the slope m. We ask: how does the cost change if we wiggle m a little? Differentiate each squared term:

Two pieces show up:

• the 2 × residual comes from differentiating the square (the derivative of x² is 2x),
• the −xᵢ comes from the chain rule: inside the bracket, the only part that depends on m is −m·xᵢ, whose derivative with respect to m is −xᵢ.

Tidying up:

Nudging the intercept c. Same logic, but now the only part inside the bracket that depends on c is +c, whose derivative is 1, so the chain-rule factor is just −1:

So the two gradients you drop into the update rule are exactly:

3. Intuition behind gradient descent

Imagine standing on a mountain and trying to find the lowest point.

With every iteration:

• Compute the direction and magnitude in which you want to move in order to reduce the loss and make your parameters better.
• Thus we say that with every step in the training process, the model is getting better and better and better.

Watch it happen below: the line starts from a random slope and intercept, then gradient descent nudges m and c step by step until the line settles into the best fit and the SSR bottoms out. Hit Randomise & retrain to drop a fresh random start and watch it converge again.

The view above watches the line move through the data. The view below watches the same process from the other side: the loss landscape, where the ball rolls downhill and the learning rate η sets the step size.

The learning rate η controls the step size: we don't want to move too fast (overshooting) or too slow (taking forever to converge).

4. How SSR minimization leads to optimal coefficients

• High SSR means the line is a bad fit (large prediction errors).
• Gradient descent tweaks m and c iteratively to reduce SSR.
• As SSR decreases, the regression line gets closer to the actual data points.
• Eventually, when the gradients become small enough, we've reached the best possible m and c.

5. Alternative: normal equation

Instead of using gradient descent, there's a direct mathematical way to find optimal coefficients. In matrix form, with design matrix X and target vector Y, and parameter vector θ (here θ = [c, m]ᵀ when the model is y = mx + c in column form):

Implementing linear regression from scratch

Before the full multi-feature version, here's the gradient-descent loop in its simplest form — walk it stage by stage. Press Run to compile the code in your browser and watch the line glide onto the data as the matching lines light up.

import numpy as np # 1. the data — features X, targets yX = np.array([0.5, 1.3, 2.1, ... , 9.4])y = np.array([2.1, 2.5, 3.0, ... , 8.6])X = (X - X.mean()) / X.std()          # standardize # 2. initialize the parametersw, b = 0.0, 0.0lr, epochs = 0.15, 80n = len(X) for epoch in range(epochs):    # 3. forward pass — predict    y_pred = w * X + b    # 4. residuals    error = y_pred - y    # 5. loss — mean squared error    loss = np.mean(error ** 2)    # 6. gradients    dw = (2/n) * np.sum(error * X)    db = (2/n) * np.sum(error)    # 7. update — step down the gradient    w -= lr * dw    b -= lr * db

The playground above fit a toy line. Now let's scale up to ten features, and rather than paste code you carry off to a notebook, run and tweak the whole thing right here.

To prove the algorithm actually works, we plant a known answer: we build a dataset of 442 samples and 10 features from a fixed set of "true" weights plus noise, then check whether gradient descent can rediscover those weights from the data alone. The train/test split, the loop, and the gradients we just derived are all hand-written NumPy.

Run it. Within a few hundred epochs the learned weights snap onto the planted true_w (including the two zeros, the features that genuinely carry no signal), the bias lands near 1.5, and the test MSE settles near the noise floor of 0.25. That is the whole of linear regression: the same residuals, the same gradients, the same update rule you watched animate at the top of this lesson, now learning eleven parameters from ten-dimensional data with nothing but NumPy.

Now make it yours. Set lr = 1.0 and watch the loss blow up to nan (steps too big); drop it to 0.05 and watch convergence crawl. Push noise_level to 5 and see the recovered weights drift away from the truth. Break it, then fix it.

Practice it yourself

You watched the line fit itself. Now build it from scratch in the browser, with real Python and hidden tests, no setup. These four Easy drills retrace the lesson end to end — predict → residual → SSR → one step — so every piece of gradient descent passes through your own fingers.