Population Stability Index (PSI)

Background

A trained model decays the moment it ships: features drift, label distributions shift, the world moves on. The Population Stability Index (PSI) is the canonical scalar that detects this. Computed per feature, daily, against the training distribution, it tells you whether the serving population has wandered far enough that the model's calibration can no longer be trusted. Thresholds in industry are tight: $\text{PSI} < 0.1$ is no drift, $0.1 - 0.25$ is moderate, $> 0.25$ triggers a retrain.

Problem statement

Implement psi(expected, actual, n_bins=10, eps=1e-10) returning the PSI between two samples of a continuous feature:

$$\text{PSI} \;=\; \sum_{i=1}^{M}\, (q_i - p_i)\, \ln\!\frac{q_i}{p_i}$$

where $p_i$ is the fraction of the expected (reference) sample in bin $i$ and $q_i$ is the fraction of the actual (serving) sample in bin $i$. Bins are equal-width over the expected sample's $[\min, \max]$. Values outside that range are clipped into the edge bins so the divergence still computes.

A small $\epsilon$ is added to each fraction so that empty bins do not produce $\ln 0$ or $0/0$.

Input

• expected - array-like of floats, the reference (training-time) sample.
• actual - array-like of floats, the new (serving-time) sample.
• n_bins - int, number of equal-width bins on the expected range (default $10$).
• eps - float, small constant added to fractions to stabilise the log (default $10^{-10}$).

Output

• float >= 0. Standard interpretation: $<0.1$ no drift, $0.1-0.25$ moderate, $>0.25$ significant.

Examples

Example 1 - identical distributions -> PSI ~= 0

Input:  expected = [0.0, 0.1, ..., 0.9], actual = expected, n_bins = 10
Output: ~0.0

Example 2 - hand-computed two-bin shift

Input:  expected = [0.1, 0.4, 0.6, 0.9],
        actual   = [0.2, 0.3, 0.4, 0.7],
        n_bins = 2
Output: ~0.2747

Explanation: bins are $[0.1, 0.5)$ and $[0.5, 0.9]$. $p = [0.5, 0.5]$; $q = [0.75, 0.25]$. $\text{PSI} = 0.25 \ln(1.5) + (-0.25)\ln(0.5) \approx 0.275$.

Example 3 - degenerate expected (all the same value) -> PSI = 0

Input:  expected = [5, 5, 5, 5], actual = [0, 5, 10, 100], n_bins = 5
Output: 0.0

Explanation: with no range in expected there are no meaningful bins, so the function returns $0$ rather than $\infty$.

Constraints

• Bins are equal-width on [expected.min(), expected.max()]. Values in actual outside that range are clipped into the edge bins.
• Result must be finite (no inf, no nan); the eps stabiliser is what guarantees this.
• Return a plain Python float. Tests compare with atol ~= 1e-3.

Notes

• Why equal-width and not equal-mass. Equal-width on the expected range is the canonical industry choice; it makes the score comparable across time. Equal-mass (quantile bins) is a known variant but masks drift in sparse regions.
• What PSI doesn't catch. Feature distributions can be stable while the relationship $P(y \mid x)$ moves (concept drift). PSI is a feature-drift detector; concept drift needs a held-out labelled stream re-scored periodically.