Risk-Coverage Curve (AURC)

Background

Selective prediction lets a model abstain on low-confidence inputs. The risk-coverage curve plots the error rate (risk) against the fraction of inputs answered (coverage); AURC (area under risk-coverage) is the scalar summary — lower is better.

Problem statement

Implement aurc(confidences, correctness). Sort items by confidence descending. For each prefix length $k$, compute the average error rate over the top-$k$ items; return the mean of those values across all $k$.

$$\text{risk}(k) \;=\; \frac{1}{k}\sum_{i \le k} (1 - y_i),\quad \text{AURC} \;=\; \frac{1}{n}\sum_{k=1}^{n} \text{risk}(k)$$

Input

• confidences — array-like of floats, the model's confidence per item.
• correctness — parallel array of $\{0, 1\}$, where $1$ = right answer.

Output

Returns a Python float in $[0, 1]$.

Examples

Example 1 — all correct → AURC = 0

Input:  confidences=[0.9, 0.8, 0.7], correctness=[1, 1, 1]
Output: 0.0

Example 2 — all wrong → AURC = 1

Input:  confidences=[0.9, 0.8, 0.7], correctness=[0, 0, 0]
Output: 1.0

Example 3 — hand-computed mixed case

Input:  confidences=[0.9, 0.5, 0.3], correctness=[1, 0, 1]
Output: (0 + 0.5 + 1/3) / 3 ≈ 0.2778

Explanation: sorted by conf desc: $(0.9,1), (0.5,0), (0.3,1)$. Risk at $k=1$: $0$. $k=2$: $0.5$. $k=3$: $1/3$. Mean: $\approx 0.278$.

Constraints

• Lengths must match.
• Confidences and correctness are sorted jointly by confidence descending.
• Returns a plain Python float.

Notes

• Why AURC, not max-coverage threshold. AURC integrates over all operating points; threshold-based metrics depend on the arbitrary cut-off.
• Lower is better. A perfect selective classifier has AURC = 0; a random one has AURC ≈ overall error rate.