Multiple-Testing Correction (Bonferroni / BH-FDR)

Background

Run 50 A/B tests at $\alpha = 0.05$ and you expect $\approx 2.5$ false positives per round even when nothing has changed. Multiple-testing correction prevents this. The two most-cited methods: Bonferroni controls the family-wise error rate by scaling $\alpha$ down; Benjamini-Hochberg (BH) controls the false discovery rate by sorting p-values and using a step-up procedure.

Problem statement

Implement multiple_testing_correct(pvals, method="bh", alpha=0.05) returning a list of bool rejection decisions, one per input p-value, in input order.

Bonferroni: reject $i$ iff $p_i \le \alpha/m$, where $m =$ number of tests.

Benjamini-Hochberg: sort p-values ascending. Reject test at sorted position $k$ (1-indexed) iff $p_{(k)} \le k\alpha/m$. The cutoff is the largest such $k$.

Input

• pvals - sequence of floats in $[0, 1]$.
• method - one of "bonferroni", "bh" (default "bh").
• alpha - significance level (default $0.05$).

Output

• list[bool], same length as pvals, same order.

Examples

Example 1 - Bonferroni at alpha=0.05, m=4

Input:  pvals = [0.01, 0.04, 0.20, 0.50], method="bonferroni"
Output: [True, False, False, False]    # threshold = 0.0125

Example 2 - BH at alpha=0.05, m=4

Input:  pvals = [0.01, 0.04, 0.03, 0.20], method="bh"
Sorted: [0.01, 0.03, 0.04, 0.20]; thresholds k*alpha/m = [0.0125, 0.025, 0.0375, 0.05]
        Check: 0.01<=0.0125 (yes), 0.03<=0.025 (no), 0.04<=0.0375 (no), 0.20<=0.05 (no)
        Largest k with p_(k) <= thresh: k=1. Reject all with sorted index <= 1 -> reject {0.01}.
Output: [True, False, False, False]    # only the 0.01 entry

Constraints

• Unknown method -> ValueError.
• Out-of-range p-values -> ValueError.
• BH preserves the input order in the output, regardless of sort order used internally.

Notes

• Bonferroni is conservative. It controls the probability of any false positive, so it's stricter than BH which controls the expected fraction of false positives among rejections.
• BH variants. BHY (Benjamini-Hochberg-Yekutieli) handles dependence; the simple BH assumes independence or positive dependence and is the production default.