Advanced65 minLLM Evaluation

Build an Evaluation Pipeline

Build a golden-set eval with an LLM-as-judge scorer and a regression gate that runs in CI.

Scenario

You're shipping prompt changes weekly and regressions keep slipping through. Build an eval pipeline that gates deploys.

Objective

Implement run_eval(golden_set, prompt_version) -> Report with faithfulness and answer_relevance scores.

Starter code

Implement the TODOs to complete the lab.

eval.pypython

def run_eval(golden_set: list[dict], prompt_version: str) -> dict:
    """Run eval against golden set.
    - Generate answers with the prompt version
    - Score with LLM-as-judge on faithfulness + relevance
    - Compare to baseline; gate if regression > 5%
    """
    # TODO
    pass

def faithfulness_score(answer: str, context: list[str]) -> float:
    """LLM-as-judge: is every claim in answer supported by context?"""
    # TODO
    pass

Solution hints

1Judge model should differ from system model
2Faithfulness: every claim must be supported by context
3Answer relevance: does the answer address the question
4Gate if score drops > 5% from baseline

Validation steps

Your implementation should pass these checks.

Eval returns per-example and aggregate scores
Faithfulness is 1.0 for fully-supported answers
Regression gate triggers on > 5% drop
Eval runs in < 5 minutes for 50 examples

Run validation

This is a simulated validation environment. In production, this would run your code against the validation steps.