Brier Score + Reliability Decomposition

Background

The Brier score is the mean squared error of probabilistic predictions: $\frac{1}{n}\sum (p_i - y_i)^2$. Decompose into reliability (calibration error) and resolution (sharpness) per bin to reason about which kind of miscalibration dominates.

Problem statement

Implement brier_score(probs, labels, n_bins=10) returning (brier, reliability, resolution).

$$\text{Brier} \;=\; \frac{1}{n}\sum (p_i - y_i)^2,\quad \text{Rel} \;=\; \frac{1}{n}\sum_m n_m\,(\bar{p}_m - \bar{y}_m)^2,\quad \text{Res} \;=\; \frac{1}{n}\sum_m n_m\,(\bar{y}_m - \bar{y})^2$$

Input

• probs - array-like of floats in $[0, 1]$, length $n$.
• labels - array-like of $\{0, 1\}$, length $n$.
• n_bins - bins for the decomposition (default $10$).

Output

• (brier, reliability, resolution) - tuple of plain Python floats.

Examples

Input: probs=[0.9, 0.1, 0.9, 0.1], labels=[1, 0, 1, 0], n_bins=2
Output: brier=0.01, reliability~0.01, resolution=0.25

Constraints

• Equal-width bins on $[0, 1]$ with clamp at the last bin.
• Empty bins contribute 0 to both reliability and resolution.

Notes

• Identity. Brier = Rel - Res + Unc (uncertainty = $\bar{y}(1-\bar{y})$). Rel = 0 means perfectly calibrated; Res large means the bins differentiate outcomes well.