Two-Sample Test + p-value

Background

Every offline A/B decision lands on the same scalar: a $p$-value from a two-sample test. The Welch / large-sample form is the workhorse — it does not assume equal variances between arms, which is the canonical L4 mistake of using a pooled Student-$t$ when the variances differ. For sample sizes typical of online experiments (thousands or more per arm) the normal approximation is exact enough that practitioners report a $z$-statistic and a two-sided $p$-value.

Problem statement

Implement two_sample_test(x, y) returning $(z, p)$ for the difference in means between two independent samples:

$$z \;=\; \frac{\bar{x} - \bar{y}}{\sqrt{\,s_x^2/n_x \;+\; s_y^2/n_y\,}}, \qquad p \;=\; 2\bigl(1 - \Phi(|z|)\bigr)$$

where $\bar{x}, \bar{y}$ are the sample means, $s_x^2, s_y^2$ are the sample variances (Bessel-corrected, divisor $n - 1$), and $\Phi$ is the standard-normal CDF.

Input

• x - array-like of floats, sample from arm A. Must have $\ge 2$ elements.
• y - array-like of floats, sample from arm B. Must have $\ge 2$ elements.

Output

• (z, p) - tuple of plain Python floats. z may be negative; p lies in $[0, 1]$ (two-sided).

Examples

Example 1 - clear separation between arms

Input:  x = [1, 2, 3, 4, 5], y = [6, 7, 8, 9, 10]
Output: (z, p) ~= (-5.0, 5.73e-7)

Explanation: $\bar{x}=3$, $\bar{y}=8$, $s_x^2=s_y^2=2.5$, $\text{SE} = \sqrt{0.5 + 0.5} = 1$. $z = (3-8)/1 = -5$. $p = 2(1 - \Phi(5)) \approx 5.73 \times 10^{-7}$.

Example 2 - identical samples

Input:  x = [1, 2, 3], y = [1, 2, 3]
Output: (0.0, 1.0)

Constraints

• Use the sample variance ($n - 1$ in the denominator), not the population variance.
• Two-sided $p$-value: $p = 2(1 - \Phi(|z|))$, computed from math.erf ($\Phi(z) = \tfrac{1}{2}(1 + \mathrm{erf}(z/\sqrt{2}))$).
• Raise ValueError if either sample has fewer than 2 elements (sample variance undefined).
• Tests use atol ~ 1e-6 on $z$ and rtol ~ 1e-3 on $p$.

Notes

• Why Welch's form (unpooled SE). Pooled-variance Student-$t$ assumes equal variances. In A/B-test data the variances of treatment and control differ routinely (a feature change that shifts watch-time means almost always shifts its variance too). Welch's form drops the assumption at no cost.
• When the normal approximation breaks. For $n < ~30$ per arm, prefer the Welch-$t$ with its degrees-of-freedom approximation. At experimentation scale this is rarely the binding constraint.