Deep Neural Networks

Lesson 3

Activation functions

Why ReLU, sigmoid, and softmax give networks their expressive power

Activation functions, ReLU, sigmoid, softmax

An activation function takes the pre-activation z (the output of Wx + b) and produces the value a neuron passes forward. Without nonlinear activations in hidden layers, a deep stack would be no more expressive than one thin linear map. With them, each layer can bend, gate, and recombine information, this is where depth earns its keep.

This lesson focuses on three workhorses: ReLU (default in many hidden layers), sigmoid (smooth squashing, classic in binary heads), and softmax (turn a vector of logits into probabilities over classes).

If activations were never there (only linear)

Compose two linear layers:

W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2)

Still linear in x. Any depth of pure linear transforms collapses to one matrix and one bias. The network could not learn XOR-style boundaries or hierarchical features that need curved decision surfaces.

Without nonlinearity in hidden layers

Depth buys nothing beyond a single linear model, same limitations as plain linear / logistic regression on raw features (for standard dense nets).

When they are present: each layer can implement new distortions of space; later layers act on features that earlier layers invented. Training adjusts W, b so those distortions help the loss go down.

ReLU (Rectified Linear Unit)

\text{ReLU}(z) = \max(0, z)

Behavior: z negative → output 0; z positive → output z (identity).

Why it helps

Cheap to compute; sparse activations (many zeros)
Gradient is 1 when z > 0, eases training of deep nets vs saturated sigmoids

Watch-outs

If a unit always stays ≤ 0, gradient is 0, “dead neuron.” Variants (Leaky ReLU, GELU, …) soften this.

Typical role: hidden layers in CNNs and MLPs (popularized in part through work associated with Geoffrey Hinton’s group on deep supervised learning).

Shape intuition (piecewise linear): flat at zero, slope 1 for positive inputs, like a hinge that clips negative mass.

Sigmoid

\sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1)

Behavior: smooth S-curve; squashes ℝ into (0, 1).

Useful for…	Caveat in deep hidden stacks
Binary output or probability of one event (logistic neuron)	For large \|z\|, derivative ≈ 0 → vanishing gradients through many layers
So: sigmoid is still everywhere in teaching and output heads; ReLU-like functions became the default between hidden layers partly for gradient flow.

Softmax

Input a vector of K logits z; output K nonnegative numbers that sum to 1:

\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

Behavior: amplifies the largest logits (relative to others); competitive normalization across classes.

Typical role: final layer for multi-class classification (one true label), paired with cross-entropy loss.

Not the same as K independent sigmoids: softmax couples all K outputs (probabilities compete to sum to 1). For multi-label problems (several classes can be on at once), sigmoid per class is more common.

Softmax turns 3 logits into competing probabilities

z = 2.0

0.66

z = 1.0

0.24

z = 0.1

0.10

The three outputs are non-negative and sum to 1 — raise one logit and the others must give up probability.

Side-by-side

σ(z) = 1 / (1 + e⁻ᶻ)

input z0.00 → 0.5000

−50+5

The function tells you the forward value; the derivative tells you whether gradients can flow back. Drag the z-marker into the red saturation zones and watch f′(z)→0 — then use "stack N layers" to see a gradient vanish.

f(z)f′(z)saturation

Activation

z marker0.00

f(z) = 0.500
f′(z) = 0.250

Stack N layers1

0.2500

(f′ at this z)^N — the factor backprop multiplies through N layers

Healthy gradient region — backprop flows. Slide z out toward ±6.

Function	Input → output	Common placement
ReLU	Scalar z → max(0,z)	Hidden layers (element-wise on vector)
Sigmoid	Scalar z → (0, 1)	Binary head; multi-label logits; teaching
Softmax	Vector z → simplex (K probs, sum 1)	Multi-class output layer

Can different layers use different activations?

Yes. The architecture is yours to specify:

Hidden: ReLU (or GELU, Swish, …) in blocks 1–L−1
Output: softmax for 10-way MNIST, or sigmoid for binary, or identity for regression

You can even mix within a research design (e.g. tanh in an early layer, ReLU later), frameworks allow it. Practical defaults exist because some combinations train more reliably.

Rule of thumb: nonlinear activations in hidden layers; output activation (or none) matches the loss (softmax + CE, sigmoid + BCE, linear + MSE).

“More and more” activation types per layer?

You usually pick one scalar activation per layer (applied element-wise to the whole hidden vector), not a different named function for each neuron unless you design something exotic. Depth adds more applications of the same family (many ReLUs), not necessarily more names on one forward pass.

Quick reference (formulas)

\text{ReLU}(z) = \max(0,z), \qquad \sigma(z) = \frac{1}{1+e^{-z}}, \qquad \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}

Takeaway

Question	Short answer
What do activations add?	Nonlinearity so depth ≠ one linear map
If never present (hidden)?	No composition of nonlinear features
ReLU	Fast, sparse; default hidden nonlinearity in many nets
Sigmoid	Bounded scalar; binary / multi-label; vanishing grad if stacked deep
Softmax	Class probabilities that sum to 1
Mix per layer?	Yes, match hidden vs output to the task

Next: Lesson 4 connects outputs to loss, especially sigmoid + BCE.

Compare them yourself

Press Run to push the same vector through all three activations, then watch ReLU's "dead neuron" gradient zero out for every non-positive input:

Python

First run loads the Python runtime (~10 MB) — takes ~5–10 seconds. Subsequent runs are instant.

Check your understanding

1 / 9

Why does a deep network need nonlinear activations between layers?

Practice it yourself

Activations are where depth earns its keep. Probe the failure modes and the math yourself — real Python, hidden tests, no setup.

Find the dead ReLU neuronsEasy

Flag units that never activate across a batch — gradient permanently zero.

Softmax with temperatureEasy

Logits → probabilities, with the T knob that sharpens or flattens them.

Predict the classEasy

Argmax over the output layer — why you don't even need softmax to decide.

Collapse two linear layersMedium

Prove W₂(W₁x+b₁)+b₂ is just one layer — the case for nonlinearity.