CUPED Variance Reduction

Background

CUPED (Controlled-experiment Using Pre-Experiment Data, Microsoft, 2013) is a one-line trick that buys 30-50% variance reduction in most online A/B tests. The idea: subtract from every user's experiment-period metric a scaled version of their pre-experiment baseline. Per-user variance that's common to both periods cancels; what remains is the actual treatment effect.

Problem statement

Implement cuped_adjust(y, x, x_bar=None) returning the CUPED-adjusted outcome:

$$Y_i^{\text{adj}} \;=\; Y_i \;-\; \theta\,(X_i - \bar{X}), \qquad \theta \;=\; \frac{\mathrm{Cov}(Y, X)}{\mathrm{Var}(X)}$$

where $Y$ is the experiment-period metric and $X$ is the same metric measured for the same user before the experiment. The pre-experiment mean $\bar{X}$ is taken across the supplied $X$ unless the caller passes a fixed x_bar (used when the global pre-experiment mean is known from a separate source).

Input

• y - array-like of floats, experiment-period outcomes (one per user).
• x - array-like of floats, pre-experiment covariate, same length as y.
• x_bar - optional float; if None, use mean(x).

Output

• np.ndarray of shape (n,) with the adjusted outcomes. Same length and order as y.

Examples

Example 1 - strong correlation: variance drops sharply

Input:  y = pre + experiment-period noise (cor with x ~ 0.9)
Output: y_adj has variance ~= 0.19 * var(y)    (1 - rho^2 = 1 - 0.81)

Example 2 - independent x and y: no variance change

Input:  y and x uncorrelated samples
Output: y_adj has variance ~= var(y)     (theta ~ 0)

Constraints

• Use the sample covariance and variance (Bessel-corrected, divisor $n - 1$).
• If $\mathrm{Var}(X) = 0$, return y unchanged (theta is undefined; no adjustment possible).
• Output is a numpy array; do not mutate y in place.

Notes

• Why it works. Subtracting $\theta(X - \bar{X})$ removes the user-baseline component of variance. The estimator $\bar{Y}_{\text{treat}}^{\text{adj}} - \bar{Y}_{\text{control}}^{\text{adj}}$ is unbiased for the treatment effect: $X$ is determined pre-treatment so it can't carry the treatment signal.
• Variance reduction. With correlation $\rho$ between $X$ and $Y$, $\mathrm{Var}(Y^{\text{adj}}) \approx (1 - \rho^2)\,\mathrm{Var}(Y)$. For watch-time-style metrics where $\rho \approx 0.7$, that's 50% off.
• Pre-experiment mean. Passing a fixed x_bar (the population pre-experiment mean) is the convention when sub-experiment slices are evaluated — it keeps the estimator unbiased across slices.