Build questions from the use case
Write evaluation questions as tasks a user would actually bring, including common requests, difficult cases and situations with insufficient material. Easy examples alone do not describe behaviour across the workflow.
An answer is not the only output
For some tasks, the useful result is a source. For others, it is declining an out-of-scope request or handing the issue to a person. Evaluation should recognise those outcomes instead of only checking for a long generated response.
Record the shape of a failure
Missing references, version mismatches, entity confusion and formatting errors are different problems. Recording them separately helps the team understand whether to change content, workflow or the system.
Make improvements testable again
Keep a stable question set, source version and assessment criteria, then run the checks after changes. The value is not a flattering overall score; it is visibility into what changed.
