Label-Noise Detection (Confident Learning)

Background

Labelled training sets contain mislabelled examples. Confident learning (Northcutt et al.) detects them probabilistically: rows where the model is highly confident about a class different from the assigned label are flagged for re-review.

Problem statement

Implement find_mislabels(predicted_probs, labels, threshold=0.7). For each row, take argmax of predicted_probs. If that's different from the row's label AND the model's confidence is above threshold, flag the row.

Input

• predicted_probs — np.ndarray shape $(N, K)$, each row sums to ~1.
• labels — sequence of integer class labels, length $N$.
• threshold — float in $[0, 1]$, confidence cut-off (default $0.7$).

Output

Returns a list[int] of row indices suspected of being mislabelled. Order matches input row order.

Examples

Example 1 — confident contradiction flagged

predicted_probs = [[0.05, 0.95],   # predicts class 1 with 0.95
                   [0.9,  0.1],    # predicts class 0 with 0.9
                   [0.6,  0.4]]    # predicts class 0 with only 0.6
labels = [0, 0, 0]
threshold = 0.7
Output: [0]

Explanation: row 0 strongly predicts 1 but is labelled 0 → flagged. Row 1 agrees. Row 2's confidence is below threshold.

Example 2 — threshold too tight

threshold = 0.99 on row with 0.6 confidence
Output: []   # no flags

Constraints

• predicted_probs.shape[0] == len(labels).
• Threshold strictly greater (>) — exactly-at-threshold rows are NOT flagged.
• Returns list[int] (plain Python ints).

Notes

• Probabilities must be calibrated. A poorly-calibrated model will flag every label (false positives) or none (false negatives). Pair with Platt or isotonic calibration first.
• Cross-validation, not training-set probs. In production you use out-of-fold predictions to avoid the model just memorising the labels.