Fleiss' Kappa

Background

Fleiss' kappa extends Cohen's kappa to settings where each item is rated by multiple raters (typically 3+). It measures inter-rater agreement after correcting for chance, returning a single scalar in $[-1, 1]$ where 1 = perfect agreement, 0 = chance-level, negative = systematic disagreement. Standard metric for RLHF data-quality QA and human-evaluation pipelines.

Problem statement

Implement fleiss_kappa(ratings) where ratings is shape $(N, K)$ with ratings[i, k] = number of raters who assigned item $i$ to category $k$. The total per row must be constant (equal raters per item). Compute:

$$P_i = \frac{1}{n(n-1)}\Bigl(\sum_k n_{ik}^2 - n\Bigr),\quad \bar P = \tfrac{1}{N}\sum_i P_i,\quad P_e = \sum_k p_k^2,\quad \kappa = \frac{\bar P - P_e}{1 - P_e}$$

where $n$ is the per-item rater count and $p_k = \tfrac{1}{Nn}\sum_i n_{ik}$.

Input

• ratings — np.ndarray shape $(N, K)$ of non-negative integer counts. Each row sums to the same total $n$ (the number of raters).

Output

Returns a Python float, the Fleiss kappa.

Examples

Example 1 — perfect agreement

Input:  ratings = [[3,0], [0,3], [3,0]]   # 3 raters, 2 categories, all agree per item
Output: 1.0

Example 2 — chance-level agreement

Input:  ratings drawn from uniform multinomial
Output: ≈ 0.0

Constraints

• Rows must sum to the same value (equal raters per item). ValueError otherwise.
• Need $n \ge 2$ raters per item.
• If $P_e = 1$ (only one category has any votes), return $1.0$.

Notes

• Interpretation. $\kappa \ge 0.6$ is typically called "substantial agreement"; $\ge 0.8$ "almost perfect". Below $0.4$ means the labelling task / guidelines need work.
• vs Cohen. Cohen's kappa needs exactly 2 raters; Fleiss handles any number $\ge 2$.