Dense layer with ReLU (the production hidden default)

Background

The course's worked net uses sigmoid in the hidden layer for uniform math, but in practice hidden layers use ReLU — it's fast, sparse, and doesn't vanish gradients for positive inputs. A ReLU dense layer is:

$$\mathbf{a} = \text{ReLU}(W\mathbf{x} + \mathbf{b}) = \max(0,\; W\mathbf{x} + \mathbf{b})$$

where W has shape (n_out, n_in).

Problem statement

Implement relu_layer(x, W, b) returning $\max(0, W\mathbf{x} + \mathbf{b})$.

Input

• x — input vector, length n_in.
• W — weight matrix (n_out, n_in).
• b — bias vector (n_out,).

Output

Returns an np.ndarray of shape (n_out,), all entries $\ge 0$.

Examples

Example 1

Input:  x = [1, -1], W = [[1, 1], [2, -1]], b = [0, 0]
Output: [0. 3.]

Explanation: $\mathbf{z} = [1-1,\; 2+1] = [0, 3]$; ReLU clips nothing here except the 0.

Example 2 — negative pre-activations clip to 0

Input:  x = [1], W = [[-2]], b = [0]
Output: [0.]

Explanation: $z = -2 \to \max(0, -2) = 0$.

Constraints

• Compute W @ x + b, then apply max(0, ·) elementwise.
• Return an np.ndarray of shape (n_out,).

Notes

• Swapping this ReLU hidden layer in (and pairing the sigmoid output with BCE) is the standard production upgrade over the all-sigmoid teaching net.