Cost-Sensitive Decision Threshold

Background

In fraud, lending, and medical screening, false positives and false negatives have different dollar costs. The optimal decision threshold isn't the one that maximises accuracy - it's the one that minimises expected cost: $c_{\text{FP}} \cdot \text{FPR} + c_{\text{FN}} \cdot \text{FNR}$. This shifts as the cost ratio changes (high-value fraud tolerates more false positives; user-friction-sensitive products do not).

Problem statement

Implement optimal_threshold(scores, labels, c_fp, c_fn). Search every unique score as a candidate threshold; for each, compute FPR and FNR; return the threshold that minimises:

$$\mathbb{E}[\text{cost}] \;=\; c_{\text{FP}}\,\text{FPR}(\theta) \;+\; c_{\text{FN}}\,\text{FNR}(\theta)$$

Return a tuple (threshold, cost).

Input

• scores - array-like of floats, model scores. Higher = more likely positive.
• labels - array-like of $\{0, 1\}$, ground truth.
• c_fp, c_fn - float > 0, costs.

Output

• (threshold, expected_cost) - tuple of plain Python floats. Threshold is the chosen cutoff; predictions are positive when $\text{score} \ge \text{threshold}$.

Examples

Example 1 - high c_fn forces a low threshold

Input:  scores=[0.1,0.3,0.6,0.8,0.9], labels=[0,1,1,1,1], c_fp=1, c_fn=100
Output: threshold low enough to classify nearly everything positive

Constraints

• $c_{\text{FP}}, c_{\text{FN}} > 0$; raise ValueError otherwise.
• Threshold candidates are the unique values of scores (a finer grid is unnecessary - cost is piecewise-constant between them).
• Tie-break by lower threshold when two candidates yield the same cost.

Notes

• Why threshold is a product decision. The cost ratio is product-defined, not learned. Multiple thresholds per segment (high-value vs low-value transactions, premium vs free users) is common.
• Calibration matters. If the score isn't calibrated, the "optimal" threshold drifts with the score distribution. Pair this with Platt scaling.