Deep Neural Networks

Lesson 1

From linear regression to the perceptron

The smallest unit: a weighted sum, a bias, and an activation

Deep neural networks are built from layers of simple units. Before depth or convolutions, it pays to understand the smallest piece: one weighted sum of inputs, plus bias, optionally passed through an activation. This lesson walks from linear regression → logistic regression → perceptron, then clarifies what people mean by a neuron today, and how that story fits the connectionist view researchers like Geoffrey Hinton championed: intelligence from many adjustable units learning from data, not hand-written rules alone.

One picture for the whole lesson

Features x → linear part z = wᵀx + b → activation f(z) → output

Change only f and the loss, you move from regression to classification to a modern building block.

One neuron: three weighted inputs, a bias, and the sigmoid activation give a binary prediction ŷ.

+1−1decision line

w₁1.00

w₂1.00

bias b0.00

z = 1.00·x₁ + 1.00·x₂ + 0.00

0.0%

accuracy on the points

Step 0, Features and a linear score

Given input vector x (features) and weights w plus bias b, the affine (linear-with-bias) map is:

z = \mathbf{w}^\top \mathbf{x} + b

This z is a score: a single number summarizing the input in a direction learned from data. Everything below reuses this same z; only what happens after z changes.

1 · Linear regression (no activation on the prediction)

Goal: predict a real value ŷ.

\hat{y} = z = \mathbf{w}^\top \mathbf{x} + b

Loss: squared error (and friends). Output is unbounded, fine for prices, temperatures, etc.

Neuron view: a “neuron” with identity activation f(z) = z. No squashing, pure linear model.

2 · Logistic regression (one neuron with sigmoid)

Goal: predict probability for binary class y ∈ 1.

Same z, then squash with the sigmoid:

\hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1)

Loss: log loss / cross-entropy. You already have one computational unit: linear + nonlinear σ, the template for a classification neuron.

Model	After z = wᵀx + b	Typical loss
Linear regression	ŷ = z (identity)	Squared error
Logistic regression	ŷ = σ(z)	Binary cross-entropy

3 · The perceptron (Rosenblatt, 1958)

Historically, the perceptron is a binary linear classifier with a hard decision:

\hat{y} = \operatorname{sign}(z) \quad\text{or}\quad \mathbb{1}[z \ge 0]

Weights were updated with the perceptron learning rule (mistake-driven). The geometry is the same linear boundary as logistic regression; the activation is not smooth, it jumps at zero.

Linear

Continuous ŷ

Logistic

Smooth σ(z)

Perceptron

Hard sign(z)

The three activations on the same score z

Linear · f(z) = z

Logistic · σ(z)

Perceptron · sign(z)

Same linear score z, three different activations, three models.

Limitation (famous): a single perceptron only learns linearly separable patterns, no XOR without multiple units / hidden layers (the start of the story for deep networks).

Perceptron vs “neuron” in modern deep learning

People use “neuron” loosely. In current frameworks, a neuron almost always means:

affine map + differentiable activation (ReLU, GELU, sigmoid, tanh, …) so gradients can flow in backpropagation.

Aspect	Classic perceptron	Typical “neuron” today
Nonlinearity	Step / sign (not differentiable at 0)	ReLU, GELU, σ, … (piecewise smooth)
Training	Perceptron rule, mistake updates	Gradient descent on a loss (backprop)
Output role	Often hard class label	Real-valued activation feeding next layer
Summary	Logistic regression is closer to a modern classification neuron (smooth σ) than the original perceptron sign, but all three share the same linear core z = wᵀx + b.

Hinton and the idea of neural computation

Geoffrey Hinton did not invent the perceptron; his influence is in treating networks of simple units as the right substrate for learning representations: adjust many w’s and b’s from examples so hierarchical features emerge. Backpropagation through smooth activations made deep stacks practical, so the “neuron” in today’s deep nets is the differentiable building block you will stack in the rest of this course.

Takeaway: linear regression = linear neuron; logistic = sigmoid neuron; perceptron = historical threshold unit on the same z. Deep networks = many such units + depth and composition, next lessons.

Your whiteboard

Sketch your own version of the neuron, annotate the parts you want to remember, or draw a comparison with the perceptron's step function. Changes live in your browser only.

✏️ Scratchpad — draw, annotate, or erase freely. Changes stay in your browser only.

Build the unit yourself

One neuron is a weighted sum, a bias, and an activation. Press Run to push the same score z through all three activations, then watch a single linear unit fail on XOR:

Python

import numpy as np

def neuron(x, w, b, activation="sigmoid"):
  z = np.dot(w, x) + b              # the shared linear score
  if activation == "identity":     # linear regression
      return z
  if activation == "sigmoid":      # logistic regression
      return 1 / (1 + np.exp(-z))
  if activation == "step":         # perceptron
      return 1.0 if z >= 0 else 0.0
  raise ValueError(activation)

x = np.array([0.5, -1.2, 2.0])       # 3 features
w = np.array([1.0, 0.5, -0.3])       # weights
b = 0.2                              # bias

z = np.dot(w, x) + b
print(f"shared score z = {z:.3f}")
for act in ["identity", "sigmoid", "step"]:
  print(f"  {act:9s} -> {neuron(x, w, b, act)}")

# One linear unit cannot separate XOR (not linearly separable)
print("\nOne perceptron on XOR:")
XOR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]
w_xor, b_xor = np.array([1.0, 1.0]), -0.5
for (a, c), target in XOR:
  pred = 1 if (w_xor @ np.array([a, c]) + b_xor) >= 0 else 0
  flag = "ok" if pred == target else "MISS"
  print(f"  in={(a, c)} target={target} pred={pred} {flag}")
print("No single straight line gets all four right -> you need a hidden layer.")

First run loads the Python runtime (~10 MB) — takes ~5–10 seconds. Subsequent runs are instant.

Check your understanding

1 / 9

Rosenblatt's perceptron uses a threshold (step) activation. Why does the modern artificial neuron replace it with sigmoid (or ReLU)?

Practice it yourself

One linear core, three behaviours. Build the neuron and the perceptron from scratch in the browser, with real Python and hidden tests, no setup.

The linear score z = wᵀx + bEasy

The neuron's pre-activation score, shared by all three models.

The perceptron thresholdEasy

Fire when z ≥ 0 — the hard step that has no usable gradient.

One unit, three activationsEasy

Switch identity / sigmoid / step and watch the model change.

The perceptron learning ruleMedium

Train it on AND with mistake-driven updates — and see why XOR can't work.