Users report the assistant's answers got worse this week. Nothing in your code changed. How do you investigate?

Strong answer

Pull traces for the reported examples and check retrieval first: are the right documents being retrieved? If not, look for an index or ingestion change (new documents, a failed sync, changed formatting). Check whether the provider changed the model version or defaults. Check whether the traffic mix changed (a new team with different questions). Re-run the evaluation set against current production: if scores dropped, bisect by component (retrieval only, generation only). Communicate early to the customer with what you know and when they will hear next. Afterwards, add the failing examples to the eval set and pin model versions where possible.

Weak-answer indicators

• Starts rewriting the prompt
• No traces or evals to look at
• Silence to the customer