What is LLM-as-judge? What biases does it have and how do you mitigate them?

Strong answer

LLM-as-judge uses a model to score outputs of another model for scalable evaluation. Biases: (1) Self-preference — judges prefer their own outputs; mitigate by using a DIFFERENT judge model. (2) Verbosity bias — judges prefer longer answers; mitigate with an explicit rubric. (3) Position bias — in pairwise, judges prefer the first option; mitigate by randomising position and averaging. (4) Calibration drift — judge quality changes over time; recalibrate quarterly against human ratings (target correlation > 0.7). Always log the judge's rationale, not just scores. Use temperature 0 for determinism. Have humans review judge disagreements — both learn.

Weak-answer indicators

• Doesn't mention self-preference
• Suggests using the same model as judge
• Skips calibration
• No mention of position bias in pairwise