Reliability

Structured Output Validation

Problem

LLMs produce malformed JSON, extra fields, wrong types, or hallucinate enum values. Downstream code crashes.

Pattern

Constrain decoding with a JSON schema (where supported), then validate the output with a strict schema validator. On failure, repair or retry.

Implementation

structured.pypython

from pydantic import BaseModel, ValidationError, validator

class Answer(BaseModel):
    summary: str
    sources: list[str]
    confidence: float

    @validator("confidence")
    def clamp(cls, v):
        return max(0.0, min(1.0, v))

def call_structured(query: str, context: list[str]) -> Answer:
    for attempt in range(3):
        raw = llm.chat(
            model="gpt-4o-mini",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": "Return JSON: {summary, sources, confidence}"},
                {"role": "user", "content": query},
            ],
        )
        try:
            return Answer.model_validate_json(raw)
        except ValidationError as e:
            log.warning(f"Attempt {attempt} invalid: {e}")
            continue
    raise RuntimeError("Could not produce valid structured output")

Trade-offs

Downstream code can trust the shape
Schema-constrained decoding can fail on edge cases
Validation + retry adds latency
Over-strict schemas cause retry storms

Production checklist

Pydantic / Zod schema as single source of truth
Retry on validation failure
Repair function for common malformations
Log validation failure rate
Schema versioning