LLM as judge
LLM as judge is using one language model to grade another model’s output against a written rubric.
It scales evaluation beyond what a human can read, at the cost of inheriting the judge’s own biases — judges favour longer answers and answers resembling their own style. Rubrics with concrete criteria, and judging one dimension at a time, reduce both effects.
Related terms
- Prompt evaluation
- Prompt evaluation is testing a prompt against a fixed set of inputs and grading criteria to find out whether a change actually improved it.
- Rubric
- A rubric is an explicit set of scored criteria used to judge output consistently, rather than relying on an overall impression.
- Self-consistency
- Self-consistency is a technique that samples several independent answers to the same prompt and takes the most common result, instead of trusting a single response.
More on evaluation
See the full glossary, read the guides, or put it into practice in the prompt builder.