Expected Calibration Error (ECE)

Background

A model's predicted probabilities are calibrated when they match observed frequencies — among the predictions it gives confidence 0.7, 70% should turn out to be positive. Modern neural networks are often accurate but badly miscalibrated: their AUC is great while their probabilities are systematically too high (or too low). That gap matters for downstream auctions, multi-task fusion, and A/B test interpretation — every system that uses the probability, not just the ranking, breaks when calibration drifts. Expected Calibration Error (ECE) is the standard scalar measure of that gap.

Problem statement

Implement expected_calibration_error(probs, labels, n_bins=10). Bin the predicted probabilities into n_bins equal-width bins over $[0, 1]$, then for each non-empty bin compute the absolute gap between empirical positive rate ($\text{acc}$) and average predicted probability ($\text{conf}$), and take the sample-weighted average:

$$\text{ECE} \;=\; \sum_{m=1}^{M}\; \frac{|B_m|}{n}\; \bigl|\, \text{acc}(B_m) - \text{conf}(B_m)\, \bigr|$$

where $B_m$ is the $m$-th bin, $|B_m|$ is the number of predictions in that bin, $\text{acc}(B_m)$ is the mean of the binary labels in the bin, and $\text{conf}(B_m)$ is the mean of the predicted probabilities in the bin.

Input

• probs — array-like of floats in $[0, 1]$, the predicted probability of the positive class (length $n$).
• labels — array-like of $\{0, 1\}$ ground-truth labels, same length as probs.
• n_bins — int, number of equal-width bins on $[0, 1]$ (default $10$).

Output

Returns a Python float in $[0, 1]$ — the expected calibration error. Empty bins contribute $0$ (they are skipped, not divided by zero).

Examples

Example 1 — perfectly calibrated

Input:  probs = [0.5, 0.5], labels = [0, 1], n_bins = 10
Output: 0.0

Explanation: both predictions land in bin 5. Empirical positive rate $=\tfrac{0+1}{2}=0.5$ exactly matches average confidence $0.5$, so the gap is $0$.

Example 2 — hand-computed two-bin case

Input:  probs = [0.2, 0.3, 0.7, 0.8], labels = [0, 1, 0, 1], n_bins = 2
Output: 0.25

Explanation: bin 0 has $(0.2, 0.3)$ with labels $(0, 1)$: $\text{acc}=0.5$, $\text{conf}=0.25$, contribution $\tfrac{2}{4}\cdot 0.25 = 0.125$. Bin 1 has $(0.7, 0.8)$ with labels $(0, 1)$: $\text{acc}=0.5$, $\text{conf}=0.75$, contribution $\tfrac{2}{4}\cdot 0.25 = 0.125$. Total $0.25$.

Example 3 — worst-case overconfidence

Input:  probs = [0.9, 0.9, 0.9, 0.9], labels = [0, 0, 0, 0], n_bins = 10
Output: 0.9

Explanation: all four predictions fall in bin 9. $\text{acc}=0$, $\text{conf}=0.9$, contribution $\tfrac{4}{4}\cdot 0.9 = 0.9$.

Constraints

• Bins are equal width on $[0, 1]$: bin $m$ covers $[m/M,\, (m+1)/M)$, except the last bin which is closed on the right so $\text{prob}=1.0$ lands in bin $M-1$ (no IndexError).
• Empty bins contribute $0$ — skip them rather than dividing by zero.
• The result is a plain Python float, not a 0-d numpy array.
• Tests compare with tolerance atol ≈ 1e-9.

Notes

• Why ECE and not just AUC. AUC is rank-invariant — it cannot detect the systematic over- or under-confidence ECE measures. A model can have AUC $= 0.99$ and ECE $= 0.30$ at the same time; the second breaks every downstream auction and fusion.
• Bin count is a knob. $M = 15$ is the most-cited default after the original paper (Guo et al., 2017); $M = 10$ is a common textbook choice. Adaptive (equal-mass) binning is a separate variant — out of scope here.