Gradient Boosting: One Round (Stump)

Background

Gradient Boosting fits a sequence of weak learners on the pseudo-residuals of the running prediction. For mean-squared-error regression, the pseudo-residual is simply $y - F(x)$ — fit a stump to the residuals, scale by the learning rate, add to $F$. One round.

Problem statement

Implement gbm_one_round(X, y, F, lr=0.1) for regression. Steps:

1. Compute residuals $r = y - F$.
2. Find the best single-threshold split on X[:, 0] (the only feature) that minimises the sum of squared residuals across the two halves.
3. For each side, predict the mean of r on that side.
4. Update $F \leftarrow F + \text{lr} \cdot \text{stump}(X)$.

Input

• X — np.ndarray shape $(n, 1)$, single-feature input matrix.
• y — 1-D array of length $n$, target.
• F — 1-D array of length $n$, current model output.
• lr — float, learning rate (default 0.1).

Output

Returns np.ndarray of length $n$, the new $F$.

Examples

Example 1 — reduces MSE on a separable problem

Input:  X = bimodal feature, y = signs of X, F = zeros
Output: MSE(F_new) < MSE(F)

Example 2 — lr = 0 leaves F unchanged

Input:  any X, y, F
Output: F_new = F

Constraints

• $X$ has shape $(n, 1)$.
• Try every unique value in X[:, 0] as a candidate threshold; pick the one minimising squared residual.
• Returns np.ndarray of length $n$.

Notes

• Why stumps. Depth-1 trees are the simplest possible weak learner; gradient boosting compensates for their weakness with many rounds.
• Multi-feature extension. Production GBMs sweep all features, picking the best (feature, threshold) pair per round. This kernel isolates the per-feature mechanic.