LLM-as-Judge
Using a model to score the outputs of another model for scalable evaluation.
Simple explanation
Instead of humans rating every output, you use a 'judge' LLM to score them. This scales evaluation to thousands of examples. The judge must be a DIFFERENT model from the one being scored.
Engineering explanation
LLM-as-judge uses a strong model (e.g., GPT-4o) to score outputs of a system model (e.g., GPT-4o-mini) on rubrics like faithfulness, relevance, helpfulness. Biases: self-preference (use a different judge model), verbosity (use a rubric), position (randomise). Calibrate against human ratings — correlation > 0.7 to trust.
Example
System answer: 'Refunds take 5-7 days.' Judge prompt: 'Is this answer supported by the context? Score 0-1.' Judge: 'Yes, 1.0.'
When to use it
Any eval pipeline that needs to scale beyond human review. Primary method for faithfulness, relevance and quality scoring.
Common misconception
LLM-as-judge is NOT free of bias. Judges prefer their own outputs (self-preference), prefer longer answers (verbosity), and prefer the first option in pairwise. Always mitigate.