Deep Neural Networks
Lesson 2Layers in a deep neural network
Input, hidden, and output, on one canonical i → h → o network
Layers in a deep neural network
In Lesson 1, one neuron was a single linear score plus an activation. A neural network is what you get when you stack many neurons into layers: raw inputs enter on one side, hidden layers build intermediate representations, and the output layer produces the prediction. Researchers including Geoffrey Hinton popularized the view that depth lets a network learn hierarchical features (simple patterns early, composed patterns deeper) without hand-crafting rules.
This whole course uses one canonical network so the symbols never change under you. It has exactly three layers:
- Input layer: i neurons, the feature vector x.
- One hidden layer: h neurons, activation a₁.
- Output layer: o neurons, activation a₂ (the prediction).
The two weight matrices are W₁ (size i × h, input → hidden) and W₂ (size h × o, hidden → output), with bias vectors b₁ and b₂. Activations are sigmoid σ. This lesson is about fully connected (dense) layers: every neuron in one layer connects to every neuron in the next.
The three layer roles
| Layer | What it holds | Learnable part? |
|---|---|---|
| Input (i) | Your features x (pixels, tabular values, embeddings…) | No weights here, just the signal you feed in |
| Hidden (h) | The activation a₁ = σ(W₁x + b₁), the network's internal representation | Yes, W₁, b₁ sit before it |
| Output (o) | The prediction a₂ = σ(W₂a₁ + b₂) | Yes, W₂, b₂ sit before it |
Input = where the story starts; output = what you score against the label; hidden = everything in between that makes the model more than one linear map.
Visual: the canonical fully connected stack
Each column is a layer; each pill is one neuron. In a dense net, each pill connects to all pills in the previous column (only a few lines are drawn so the picture stays readable; the rule is all-to-all).
Input · i
x (i = 3)
Hidden · h
a₁ (h = 4)
Output · o
a₂ (o = 2)
Fully connected: each violet neuron receives a weighted sum from all input neurons, and each output neuron from all hidden neurons.
Forward pass through the layers
The signal moves left to right: a linear step (weights and bias), then the sigmoid σ, once per layer (this is the full subject of Lesson 5):
The hidden activation a₁ is literally the input to the output layer. That is what "stacking layers" means: the output of one layer is the input of the next.
Zoom in: every neuron is a perceptron
Those two matrix lines can make a network look like a new kind of object. It is not. Write out one row of W₁x + b₁ and you get a weighted sum of the three inputs plus a bias, squashed by σ, which is exactly the perceptron from Lesson 1. The matrix notation is bookkeeping for doing four of them at once.
Click any violet or emerald circle below to open it up, or press Tour all 6 to walk through the whole network one unit at a time. Move the x sliders and watch every value downstream recompute.
Inside the stack — every neuron is a perceptron
Click any violet or emerald circle to open it up. The depth is new; the unit inside is not.
Notice what does not change as you move between circles: the recipe is always multiply, sum, add bias, squash. Only the incoming numbers and the weights on the edges differ. The input neurons are the exception, they hold values and own no weights, which is why the network has 6 perceptrons rather than 9. Depth is repetition of one unit, not the invention of a new one.
Where weights and biases live
Parameters sit between layers, on the edges, not "inside" the circles (the circles are just values passing through):
- W₁ (i × h) and b₁ (length h) connect input → hidden.
- W₂ (h × o) and b₂ (length o) connect hidden → output.
So a fully connected layer from n_in neurons to n_out neurons holds
(weights + one bias per output neuron).
Counting parameters (our i = 3, h = 4, o = 2 net)
| Connection | Matrix | Count |
|---|---|---|
| Input 3 → Hidden 4 | W₁ (3×4), b₁ (4) | 3×4 + 4 = 16 |
| Hidden 4 → Output 2 | W₂ (4×2), b₂ (2) | 4×2 + 2 = 10 |
Total for this tiny net: 16 + 10 = 26 trainable scalars. Wider layers and more layers multiply this quickly, that is capacity.
Count them yourself
Press Run to count the parameters of the canonical net, then change the sizes and watch the total grow:
Going deeper: what depth and width buy
Our canonical net has one hidden layer. "Deep" just means stacking more hidden layers, repeating the same linear → σ block. Each extra hidden layer adds its own W and b, and the general pattern for layer ℓ is the same idea:
More neurons per layer (width)
A bigger h gives more linear combinations at once, a richer slice of functions at each step.
More hidden layers (depth)
Compose nonlinearities into features of features. This is the heart of deep learning's hierarchy story.
Tradeoff: more parameters → more flexible but easier to overfit and costlier to train. The number and size of hidden layers is a design choice, not something the network learns.
Nonlinearity between layers
If you only stacked linear maps, the whole network would collapse to one linear map. The σ between layers breaks that: each hidden layer can bend the space so later layers see new directions. No nonlinearity in the hidden layers means no extra power from depth.
Toggle activations OFF and retrain: no matter how deep, the boundary snaps back to one straight line.
Hyperparameters vs learned weights
| You set before training (hyperparameters) | The optimizer learns from data |
|---|---|
| i, h, o, and how many hidden layers | The entries of W₁, b₁, W₂, b₂ |
| Choice of activation, learning rate, batch size | The values that minimize the loss on examples |
The activations a₁, a₂ change every step as the input changes; the sizes i, h, o are fixed unless you redesign the net.
Takeaway
| Idea | One line |
|---|---|
| Input layer | i neurons, the features x you provide |
| Hidden layer | h neurons, the learned vector a₁; W₁, b₁ sit before it |
| Output layer | o neurons, the prediction a₂; W₂, b₂ sit before it |
| Dense | Full connectivity layer to layer |
| Parameters | n_in × n_out + n_out per layer (here 26 total) |
| Depth / width | Complexity and cost knobs you set by hand |
Next lessons train this stack: activations (Lesson 3), loss (Lesson 4), and the forward and backward passes (Lessons 5–6) on this exact i → h → o network.
Check your understanding
1 / 7A dense (fully-connected) layer maps n_in = 5 inputs to n_out = 3 outputs. How many trainable parameters does it have?
Practice it yourself
Count the parameters and run the signal through the canonical i → h → o net — real Python, hidden tests, no setup.
n_in·n_out + n_out — weights plus one bias per output neuron.
Sum across layers — and see why the input layer holds no weights.
σ(Wx + b) for one layer — the block you stack to make depth.
Chain i → h → o: a₂ = σ(W₂·σ(W₁x + b₁) + b₂).