Methodology guide
Designing a useful evaluation
Define the capability a test measures, preserve its conditions, and report what its results can and cannot establish.
When to use it
Start with a clear purpose.
Use an evaluation to answer a specific development question: which constraint fails, where evidence is lost, or whether a changed workflow improves the deliverable. An aggregate score is useful only with the task definition and scoring rules beside it.
Step 01
Define observable criteria
Separate final correctness from evidence use, constraint adherence, and recovery after errors. Choose repeatable checks where possible and a review rubric where judgment remains necessary.
Step 02
Run a controlled comparison
Fix inputs, tool access, and budgets before comparing runs. Retain outputs and relevant action traces, and record failed runs alongside successful ones.
Step 03
Inspect the failure patterns
Review errors by task and criterion. Use repeated runs where variability matters, and distinguish scoring disagreements from failures in the submitted work.
Acceptance
Make the result reviewable.
Deliver enough information to repeat the procedure and interpret its limits. Report coverage, uncertainty, exclusions, and changes to the rubric; do not generalize beyond the tested setting.