Evaluation & Release Pipelines
Release prompt, model and retrieval changes like code: versioned, gated by evaluations, rolled out gradually and easy to roll back.
Evaluation & release pipelines
In an AI application, behaviour is decided by more than code. The prompt, the model and its version, retrieval settings, tool definitions and the knowledge base all change what users see. Each of these is a release, whether or not anyone treats it as one.
A release pipeline treats them like code:
Why it matters
AI changes fail quietly. A prompt tweak that fixes one complaint can break ten cases nobody was watching, and the application keeps returning well-formed, confident answers throughout. Users notice before dashboards do.
Some changes come from outside. Providers update and retire models, sometimes with little notice, and a model alias such as "latest" can change behaviour overnight. Without pinned versions and gates, you find out from your users.
A pipeline also speeds teams up. When every change is evaluated and reversible, people ship small improvements often instead of batching risky ones.
The pipeline, stage by stage
1. Registry: prompts, configurations and model identifiers live in version control or a registry, pinned to dated model versions rather than moving aliases.
2. Pull request: a fast evaluation subset (a few minutes) runs on every change, including deterministic checks such as schema validity and refusal cases.
3. Pre-release: the full evaluation suite runs against the candidate and the current baseline, with thresholds per metric and zero tolerance on critical cases.
4. Shadow: the candidate receives a copy of live traffic, but users only see the current version. Compare outputs, cost and latency.
5. Canary: a small share of real users (for example 1–5%) get the candidate, with automatic rollback if online metrics drop.
6. Ramp and monitor: increase traffic in steps; keep the previous version deployable.
7. Record: log which versions produced every response, so incidents can be traced.
A release gate
from dataclasses import dataclass
@dataclass
class Thresholds:
max_quality_drop: float = 0.01 # absolute drop allowed on the overall score
critical_pass_rate: float = 1.0 # every critical case must still pass
max_cost_increase: float = 0.10 # +10% cost per request
max_p95_latency_increase: float = 0.15
def gate(baseline: dict, candidate: dict, t: Thresholds = Thresholds()) -> tuple[bool, list[str]]:
"""baseline/candidate: {"quality", "critical_pass_rate", "cost_per_request", "p95_latency_s"}"""
reasons = []
if baseline["quality"] - candidate["quality"] > t.max_quality_drop:
reasons.append(f"quality {candidate['quality']:.3f} vs {baseline['quality']:.3f}")
if candidate["critical_pass_rate"] < t.critical_pass_rate:
reasons.append(f"critical cases {candidate['critical_pass_rate']:.0%}")
if candidate["cost_per_request"] > baseline["cost_per_request"] * (1 + t.max_cost_increase):
reasons.append("cost per request above budget")
if candidate["p95_latency_s"] > baseline["p95_latency_s"] * (1 + t.max_p95_latency_increase):
reasons.append("p95 latency above budget")
return (not reasons, reasons)
def canary_should_roll_back(online: dict) -> bool:
"""Checked every few minutes during a canary; any breach rolls back automatically."""
return (online["error_rate"] > 0.02 or online["thumbs_down_rate"] > online["baseline_thumbs_down_rate"] * 1.5
or online["guardrail_blocks"] > online["baseline_guardrail_blocks"] * 2)
# ok, why = gate(baseline_metrics, candidate_metrics)
# if not ok: fail the pipeline and print whyExperiment: change type and release strategy
Pick a kind of change and how it is released. The outcomes are illustrative of common incidents.
What to observe
The riskiest changes are the ones nobody treats as releases: prompt edits, retrieval tweaks and model aliases that update themselves. Pinning versions turns hidden changes into explicit ones, and the gate, shadow and canary stages catch problems at the cheapest point.
Release pipelines in production
Keep the fast suite fast. If pull-request evaluation takes an hour, people stop running it; keep a few minutes' worth of the most informative cases there and run the full suite before release.
Track provider retirement dates in one calendar. Test the successor model against every use case's evaluation well before the deadline, and switch through the normal pipeline rather than in a rush.
Roll back the whole bundle. A prompt written for one model version may not suit the previous model, so version prompt, model and configuration together and roll them back as a unit.
Record versions on every response and log line. When an incident happens, the first question is which prompt, model and configuration produced the bad answer, and the answer should take seconds to find.
Challenge
Your provider will retire the model your customer-support assistant uses in 60 days. The successor is cheaper and scores slightly higher overall on your evaluation, but fails two critical refund-policy cases. Plan the migration: what you change, how you test it, how you roll it out, and how you would roll it back.
Production checklist
Production checklist
0 of 9 checked
Knowledge check
Why pin a dated model version instead of using a provider alias such as "latest"?
Complete
You can now release AI changes safely: versioned, gated, gradual and reversible. This completes the AI Platform Engineering series.
Mark this chapter as complete
Track your progress and unlock the next chapter.