Permutation Feature Importance

Background

Permutation importance measures each feature's marginal contribution to a model's predictive performance: shuffle the column, rescore, and the drop in metric equals that feature's importance. Model-agnostic, works for any score function — but compute scales as $O(\text{features} \times \text{rows})$.

Problem statement

Implement permutation_importance(model, X, y, metric, seed=0):

1. Score the baseline: baseline = metric(y, model.predict(X)).
2. For each feature $i$: copy X, shuffle column $i$, score s_i = metric(y, model.predict(X_perm)), record imp_i = baseline - s_i.

Higher importance = bigger metric drop when shuffled = feature matters more.

Input

• model — object with .predict(X) -> y_hat.
• X — np.ndarray shape $(n, d)$.
• y — 1-D array of length $n$.
• metric(y_true, y_pred) — higher = better.
• seed — int, shuffle RNG seed.

Output

Returns np.ndarray of length $d$, one importance per feature.

Examples

Example 1 — important feature gets higher score

Input:  y depends only on X[:, 0]; X[:, 1] is noise
Output: imp[0] > imp[1]

Example 2 — output length matches feature count

Input:  X shape (50, 5)
Output: shape (5,)

Constraints

• Use np.random.default_rng(seed) for reproducibility.
• Original X is not mutated.
• Don't return early on negative importance — a noisy feature's score may slightly increase the metric on shuffle; report the negative value.

Notes

• Repeats. Production runs repeat the shuffle 5–20 times per feature and report the mean ± std — single-shot importance is noisy.
• Correlated features. If feature A and B are correlated, shuffling A alone may not hurt the model much (B still carries the signal). Shapley values handle this correctly but are vastly more expensive.