How do you build an evaluation set with a customer who has never done it?

Strong answer

Collect 50–200 real questions or tasks from logs, tickets or interviews, weighted towards the common and the costly. Have the customer's experts write or approve the expected answers and the source each should come from. Label a subset as critical (safety, money, legal). Define the scoring: exact match where possible, rubric-based or model-as-judge with spot checks otherwise. Run it on every change and share the results in the weekly update. Make the customer the owner of the set so it keeps growing after you leave.

Weak-answer indicators

• Uses public benchmarks only
• Writes the answers themselves without experts
• Runs it once