Evaluation Metrics

50 problems

• Accuracy scoreEasy
• PASS or FAIL?Easy
• Validate a chat messageEasy
• BERTScore (Greedy Token-Cosine F1)Medium
• Best-of-N SelectionEasy
• BLEU score (unigram)Medium
• Bradley-Terry Strength MLEHard
• Brier Score + Reliability DecompositionMedium
• Citation Attribution AssemblyMedium
• Cohen's kappa scoreMedium
• Confusion matrixEasy
• Cost-Sensitive Decision ThresholdMedium
• Dice lossEasy
• Divide dataset by feature thresholdEasy
• Early stopping on validation lossEasy
• Elo Rating UpdateEasy
• Expected Calibration Error (ECE)Medium
• Expected Reciprocal Rank (ERR)Medium
• F1 score (binary classification)Easy
• Faithfulness / Groundedness ScoreMedium
• Fleiss' KappaMedium
• Focal loss (multiclass)Medium
• IoU of bounding boxesEasy
• Isotonic Regression (PAVA)Hard
• Jaccard similarityEasy
• k-fold cross-validationEasy
• LambdaMART Lambda GradientsHard
• mAP for Object DetectionHard
• Matthews Correlation Coefficient (MCC)Easy
• Mean Average Precision (MAP)Medium
• Mean reciprocal rank (MRR)Easy
• METEOR scoreMedium
• pass@k Unbiased EstimatorMedium
• Permutation Feature ImportanceMedium
• Perplexity from log-probsEasy
• Platt ScalingMedium
• PR-AUC (Average Precision)Medium
• Precision@k and NDCG@kMedium
• Precision metricEasy
• QA Token F1 / Exact MatchMedium
• Recall@k (Retrieval)Easy
• Risk-Coverage Curve (AURC)Medium
• ROC-AUC from scratchMedium
• ROUGE-1 scoreEasy
• Self-Consistency Majority VoteEasy
• SMOTE OversamplingMedium
• Softmax (multinomial) regressionMedium
• Spearman Rank CorrelationEasy
• Tool-Call Parse + Schema ValidationMedium
• Weighted cross-entropyEasy