Shadow / Canary Rollback Decision

Background

After a canary deploy, the platform watches a few guardrail metrics and automatically rolls back if any breaches a configured threshold. The kernel: compute per-metric deltas (canary − control), check against per-metric thresholds, return a verdict with a human-readable reason.

Problem statement

Implement decide_rollback(control, canary, thresholds). Each input dict has latency_p99, error_rate, throughput. Thresholds has max_latency_delta_pct, max_error_delta_pct, max_throughput_drop_pct. Return (action, reason):

• Latency increase > max_latency_delta_pct (%) → ("rollback", "<reason>").
• Error-rate increase > max_error_delta_pct (%) → ("rollback", "<reason>").
• Throughput drop > max_throughput_drop_pct (%) → ("rollback", "<reason>").
• Otherwise → ("keep", None).

Input

• control — dict[str, float] with latency_p99, error_rate, throughput.
• canary — dict[str, float] with the same keys.
• thresholds — dict[str, float] with max_latency_delta_pct, max_error_delta_pct, max_throughput_drop_pct.

Output

Returns a tuple (action: str, reason: str | None). action is "keep" or "rollback".

Examples

Example 1 — within thresholds → keep

control = latency 100, err 0.01, tput 1000
canary  = latency 105, err 0.011, tput 990
limits  = +10% lat, +20% err, -5% tput
Output: ("keep", None)

Example 2 — latency over → rollback

canary  = latency 200 (vs 100 control)
Output: ("rollback", "latency delta 100.0% ...")

Example 3 — error rate over → rollback Example 4 — throughput drop over → rollback

Constraints

• Check guardrails in fixed order: latency, then error, then throughput. The first to trip wins.
• Reason must mention which guardrail tripped.
• Returns Python tuple of (str, str | None).

Notes

• More guardrails in production. Real platforms also track p50 latency, retry rate, downstream-service error budgets, and product-side metrics (CTR delta, purchase rate).
• Hysteresis. Production systems wait $N$ minutes of breach before triggering to avoid flapping on a single spike.