Sample Size for Statistical Power

Background

Every A/B test starts with one question: how big does each arm need to be? Underpowered experiments routinely come back with "+0.3% lift, not significant" and get shipped on vibes; rigorous teams compute the required sample size up front. The standard formula closes the loop between three knobs: the noise in the metric ($\sigma$), the minimum detectable effect ($\delta$), and the significance / power trade-off ($\alpha$, $\beta$).

Problem statement

Implement sample_size_per_arm(sigma, mde, z_alpha2=1.96, z_beta=0.8416) returning the per-arm sample size needed to detect a difference in means with the given parameters:

$$n \;=\; \frac{(z_{\alpha/2} + z_{\beta})^{2}\, \cdot\, 2\sigma^{2}}{\delta^{2}}$$

Defaults correspond to $\alpha = 0.05$ two-sided ($z_{0.025} \approx 1.96$) and $\beta = 0.20$ i.e. 80% power ($z_{0.20} \approx 0.8416$). Round up to the next integer (you cannot run fractional users).

Input

• sigma - float > 0, standard deviation of the primary metric per user.
• mde - float > 0, the minimum detectable effect (absolute difference in means you want to be able to call).
• z_alpha2 - float, critical value for the two-sided significance level. Default 1.96 (alpha = 0.05).
• z_beta - float, critical value for the power. Default 0.8416 (power = 0.80).

Output

• int >= 1 - the required per-arm sample size, rounded up.

Examples

Example 1 - the canonical "watch time" calculation

Input:  sigma = 30.0, mde = 0.3   (a 1% lift on 30 min mean)
Output: 156980

Explanation: $(1.96 + 0.8416)^2 \approx 7.849$. $n = 7.849 \cdot 2 \cdot 900 / 0.09 \approx ~156979.2$, ceil to $156{,}980$.

Example 2 - tightening the MDE quadruples n

Input:  sigma = 30.0, mde = 0.15
Output: 627918       # ~ 4 * 156980

Explanation: $n \propto 1/\delta^{2}$; halving MDE multiplies $n$ by $4$.

Example 3 - higher power (90%) needs more samples

Input:  sigma = 30.0, mde = 0.3, z_alpha2 = 1.96, z_beta = 1.2816
Output: 210160

Explanation: $(1.96 + 1.2816)^2 \approx 10.51$, so $n \approx 10.51 \cdot 2 \cdot 900 / 0.09 \approx 210{,}159. \to 210{,}064$.

Constraints

• Raise ValueError if sigma <= 0 or mde <= 0 (the formula is undefined).
• Use math.ceil to round up; return a Python int.
• The formula is for a difference-in-means test; for a difference-in-proportions test, see the variant note.

Notes

• Rule of thumb. $(z_{\alpha/2} + z_{\beta})^{2} \approx 8$ for the default $\alpha,\beta$, giving the famous $n \approx 16\sigma^{2}/\delta^{2}$. The exact value $7.849$ is what the formula uses; the rule-of-thumb $8$ is for whiteboard arithmetic.
• Bound MDE before you optimise sample size. The smallest worthwhile MDE is bounded by what the product cares about, not by what the experiment can detect. Computing $n$ for an MDE smaller than the product cares about wastes capacity.
• CUPED. Pre-experiment variance reduction shrinks $\sigma^{2}$ by $\rho^{2}$, where $\rho$ is the correlation between the pre-experiment baseline and the experiment metric. With $\rho = 0.7$, CUPED roughly halves the required $n$.