JudgmentBench
In law and other judgment-rich domains, quality can be easier to recognize than to define. Is expert judgment more like a science, where quality can be broken into explicit criteria, or more like an art, where experienced practitioners can see what works even when they cannot fully explain why? JudgmentBench asks which of two methodologies–rubrics and comparative judgments–is better suited to assess quality in the legal domain. It brings together 30 real-world legal tasks and thousands of evaluations from experienced attorneys. The study finds that comparative judgments identify differences in quality more reliably and faster than rubric-based scoring. More broadly, it offers a framework for evaluating AI in fields where quality depends on professional judgment rather than factual correctness alone.
