SCOPE：给 LLM 裁判加一层保形校准 SCOPE: Selective Conformal Optimized Pairwise LLM Judging

精读摘要

LLM 越来越多地被当作可扩展的成对评估裁判，但仍易出现校准偏差与偏见。SCOPE 框架校准一个接受阈值，保证在可交换性条件下，未弃权的判断中错误率不超过用户设定的水平 α。为提供偏差中性的不确定性信号，团队还提出双向偏好熵（BPE）：从正反两个方向询问裁判，衡量其偏好不确定性，让裁判「没把握时弃权」。 LLMs are increasingly used as scalable judges in pairwise evaluation but remain prone to miscalibration and biases. SCOPE calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level alpha. It introduces Bidirectional Preference Entropy, querying the judge from both directions for a bias-neutral uncertainty signal.