Quality

Prompt Versioning

Problem

Prompts are code. Editing a prompt in a notebook breaks production, cannot be rolled back, and cannot be evaluated across versions.

Pattern

Version prompts as code. Tag, evaluate and deploy prompt versions through the same pipeline as application code.

Implementation

prompt_versioning.pypython

from dataclasses import dataclass

@dataclass
class PromptVersion:
    id: str
    template: str
    version: str
    eval_score: float

PROMPTS = {
    "rag_answer": [
        PromptVersion("rag_answer", v1_template, "1.0.0", 0.72),
        PromptVersion("rag_answer", v2_template, "1.1.0", 0.81),  # current
    ],
}

def get_prompt(name: str, version: str = "latest") -> str:
    versions = PROMPTS[name]
    if version == "latest":
        return versions[-1]
    return next(v for v in versions if v.version == version)

# Evaluate new prompt in CI before promoting
def promote_prompt(name, candidate):
    score = run_eval(name, candidate)
    if score < current_score(name):
        raise EvalRegressionError(score)
    PROMPTS[name].append(candidate)

Trade-offs

Safe prompt iteration
Reproducible evals across versions
Rollback on regression
Overhead of versioning and eval pipeline

Production checklist

Prompts in version control
Eval gate before promotion
Canary rollout of new prompt
Metrics per prompt version
Rollback within minutes