BERTScore (Greedy Token-Cosine F1)

Background

BERTScore evaluates text-generation quality by computing greedy token-level cosine alignment between candidate and reference embeddings, returning precision, recall, and F1 in $[-1, 1]$. Strictly better than BLEU/ROUGE on paraphrasing because it works in semantic space rather than surface-form n-grams.

Problem statement

Implement bertscore(cand_embs, ref_embs). Both are L2-normalised token-embedding matrices of shape $(T, d)$. Compute the cosine similarity matrix $S \in \mathbb{R}^{T_c \times T_r}$, then:

$$P = \frac{1}{T_c}\sum_i \max_j S_{ij},\quad R = \frac{1}{T_r}\sum_j \max_i S_{ij},\quad F_1 = \frac{2 P R}{P + R}$$

Input

• cand_embs — np.ndarray shape $(T_c, d)$, L2-normalised candidate token embeddings.
• ref_embs — np.ndarray shape $(T_r, d)$, L2-normalised reference token embeddings.

Output

Returns a tuple (P: float, R: float, F1: float) — all in $[-1, 1]$.

Examples

Example 1 — identical sequences → F1 = 1

Input:  cand_embs = ref_embs = I_3
Output: (1.0, 1.0, 1.0)

Example 2 — orthogonal sequences → F1 < 0.5

Input:  orthogonal embeddings
Output: low cosine maxes → low F1

Constraints

• Inputs must be L2-normalised (the metric assumes unit-norm vectors).
• If $P + R = 0$, return F1 = 0 (avoid division-by-zero).
• Plain Python floats.

Notes

• IDF weighting. Production BERTScore weights each reference token by IDF so rare tokens dominate. This version uses uniform weights.
• Choice of model. Empirical BERTScore depends heavily on the embedding model; published numbers use a fixed backbone (typically roberta-large).