Huber & Hinge losses

Background

Two robust margin/regression losses. Huber loss blends squared error (for small residuals) with absolute error (for large ones), so a few outliers don't dominate the gradient — the standard robust regression loss. Hinge loss is the SVM loss: it penalises predictions that are correct but not confident, enforcing a margin.

Problem statement

Implement two functions.

huber_loss(pred, target, delta=1.0) — the mean over elements of, with $r = \text{pred} - \text{target}$:

$$\ell_\delta(r) = \begin{cases} \tfrac12 r^2 & |r| \le \delta \\[2pt] \delta\,(|r| - \tfrac12\delta) & |r| > \delta \end{cases}$$

hinge_loss(pred, target) — the mean of $\max(0,\, 1 - y_i\,\hat{y}_i)$ with labels $y \in \{-1, +1\}$:

$$\mathcal{L}_{\text{hinge}} = \frac{1}{N}\sum_i \max\big(0,\; 1 - y_i\,\hat{y}_i\big)$$

Input

• pred — np.ndarray: predictions (raw scores for hinge).
• target — np.ndarray: targets (real values for Huber; $\{-1, +1\}$ labels for hinge).
• delta — float (Huber only): the transition point between the quadratic and linear regimes.

Output

Each function returns a float: the mean loss.

Examples

Example 1 — Huber

Input:  huber_loss([1, 2, 3], [1.5, 4, 3], delta=1.0)
Output: 0.5417

Explanation: residuals $[-0.5, -2, 0]$. The small one is quadratic $\tfrac12(0.5)^2=0.125$; the large one is linear $1\cdot(2-0.5)=1.5$; the zero contributes 0. Mean $=1.625/3=0.5417$.

Example 2 — Hinge

Input:  hinge_loss([0.8, -0.5, 2], [1, -1, 1])
Output: 0.2333

Explanation: margins $y\hat{y}=[0.8, 0.5, 2]$, so losses $\max(0, 1-\cdot)=[0.2, 0.5, 0]$ and the mean is $0.7/3=0.2333$.

Constraints

• Huber is quadratic when $|r|\le\delta$ and linear ($\delta(|r|-\tfrac12\delta)$) beyond — the two pieces meet smoothly at $\pm\delta$.
• Hinge labels are $\{-1, +1\}$; points correctly classified beyond the margin ($y\hat{y}\ge 1$) contribute 0.
• Both functions return the mean over elements.

Notes

• As $\delta\to\infty$ Huber becomes MSE; as $\delta\to 0$ it becomes (scaled) MAE — it interpolates between the two.
• Hinge is non-smooth at the margin; its sub-gradient is $-y_i x_i$ when $y\hat{y}<1$ and $0$ otherwise, exactly what the SVM/Pegasos update uses.