Sample Ratio Mismatch (SRM) Check

Background

Sample Ratio Mismatch (SRM) is the #1 silent failure of online experiments. You expect a 50/50 traffic split; you see 48.7/51.3. With enough users, that 2.6% drift is wildly statistically significant - and it means something is broken upstream (assignment bug, caching, redirect-loss bias). Production experimentation platforms run an SRM chi-square on every experiment automatically; if it fires, the experiment is invalid and the metric numbers are not to be trusted.

Problem statement

Implement srm_check(arm_counts, expected_proportions, alpha=0.001) running a chi-square goodness-of-fit test:

$$\chi^2 \;=\; \sum_{k}\, \frac{(O_k - E_k)^2}{E_k},\qquad p = 1 - F_{\chi^2_{K-1}}(\chi^2)$$

with $K$ arms, $O_k$ observed counts, $E_k = N \cdot \pi_k$ expected counts. Return (chi2, p_value, srm_detected).

Input

• arm_counts - sequence of int, observed user counts per arm.
• expected_proportions - sequence of float summing to $1$, the planned split.
• alpha - significance level (default $0.001$; SRM is treated as serious, so a tight $\alpha$).

Output

• (chi2, p_value, srm_detected) - chi-square statistic (float), p-value (float in $[0,1]$), boolean.

Examples

Example 1 - clean 50/50

Input:  arm_counts=[5000, 5000], expected_proportions=[0.5, 0.5]
Output: (0.0, 1.0, False)

Example 2 - SRM fires

Input:  arm_counts=[4870, 5130], expected_proportions=[0.5, 0.5]
Output: chi2 ~ 6.76, p ~ 0.0093, srm_detected = True at alpha=0.05

Constraints

• arm_counts and expected_proportions must have the same length and $\ge 2$ arms.
• expected_proportions must sum to $1$ (within $10^{-6}$).
• Use scipy.stats.chi2.sf(stat, df=K-1) or the closed-form via math.exp etc; for $K \le 3$ a numpy-only implementation is also OK.
• Return Python floats and a bool.

Notes

• Why SRM matters more than a "small effect." If users are assigned to the wrong arms, every metric on the experiment is biased - including the metrics you're trying to read. SRM is an integrity check, not a power check.
• Alpha is tight. Production platforms use $\alpha = 0.001$ because false-positive SRMs are very expensive (you re-investigate the platform every time).