Bayesian A/B Test (Beta-Binomial)

Background

Bayesian A/B testing replaces "is the p-value below 0.05?" with the directly interpretable "what's the probability that B is better than A?" The Beta-Binomial model is the canonical choice for conversion-rate metrics: with conjugate Beta priors and Binomial likelihoods, the posterior for each arm's true rate is Beta-shaped, and the probability one arm exceeds the other is estimated by sampling from the two posteriors and counting.

Problem statement

Implement bayesian_ab_prob(a_succ, a_n, b_succ, b_n, prior_alpha=1.0, prior_beta=1.0, n_samples=20000, seed=0) returning $P(\theta_B > \theta_A \mid \text{data})$.

Posteriors (conjugate Beta-Binomial):

$$\theta_A \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + s_A,\ \beta_0 + n_A - s_A), \qquad \theta_B \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + s_B,\ \beta_0 + n_B - s_B)$$

Estimate the desired probability by Monte Carlo: draw n_samples from each posterior independently and report the fraction of draws where the B-sample exceeds the A-sample.

Input

• a_succ - int >= 0, successes in arm A.
• a_n - int >= a_succ, total trials in arm A.
• b_succ, b_n - same for arm B.
• prior_alpha, prior_beta - float > 0, hyperparameters of the Beta prior (default $\mathrm{Beta}(1, 1)$, the uniform).
• n_samples - int, Monte Carlo samples per arm (default $20{,}000$).
• seed - int, RNG seed (default $0$). Required for reproducibility in tests.

Output

• float in $[0, 1]$ - the estimated $P(\theta_B > \theta_A)$.

Examples

Example 1 - B clearly better

Input:  a_succ=10, a_n=100, b_succ=20, b_n=100
Output: >0.99

Example 2 - Equal evidence, same priors

Input:  a_succ=50, a_n=100, b_succ=50, b_n=100
Output: ~0.5

Constraints

• Use a single RNG seeded by seed so the test is reproducible.
• Raise ValueError if a_succ > a_n or b_succ > b_n (impossible counts).
• Return a Python float.

Notes

• Why Bayesian over frequentist. $P(B > A)$ is the question stakeholders ask; the $p$-value is a question about the data under a null. Bayesian framing also lets you state "we have 92% probability that B is at least 1% better" with no $\alpha$-$\beta$ trade-off.
• Prior choice. $\mathrm{Beta}(1, 1)$ is uniform (no opinion). Stronger priors ($\mathrm{Beta}(\alpha, \alpha)$ with larger $\alpha$) shrink the posterior toward $0.5$ - useful for high-variance / low-traffic settings.
• Closed form. There is a closed-form expression for $P(\theta_B > \theta_A)$ as a sum of hypergeometric-flavoured terms, but Monte Carlo is the production default because it generalises to non-conjugate priors with no re-derivation.