Glossary Evaluation set
Evaluation set
Held-out tasks used to measure a stated agent capability, with the scoring method agreed before collection starts.
Answer in brief
An evaluation set is a group of held-out tasks used to measure one stated agent capability. Each item should answer a concrete question: whether the agent routes to the right queue, extracts the right field, or stops before an irreversible action.
An aggregate accuracy figure is where evaluation discussions usually end and where most of the useful information is lost. Ask for the failure slices, the item count per slice, and the versions of the model, prompt, dataset and scorer.
Definition
An evaluation set is a collection of tasks held out from training, used to measure a stated capability under a scoring method agreed in advance. The stated capability is part of the definition: a set that does not name what it measures is a sample, not an evaluation.
The set is only as good as its hold-out. If the same items appear in the delivered training data, or the supplier tuned against them, the score describes memory rather than capability. The guide to evaluation sets takes this further; the rubric entry covers the scoring rules.
One item, one question
Write down the question each item answers before writing the item. A useful question is narrow enough that two reviewers agree on the expected outcome: does the agent stop when the record is locked, retry with a corrected argument, escalate when a permission check fails.
Group items into failure slices — routing, extraction, policy compliance, recovery after a tool error, escalation — and ask for per-slice counts, so a thin slice cannot hide behind a strong average.
Hold-out hygiene
Hold-out hygiene has to survive the whole engagement. Ask how evaluation items are separated from training data, whether the supplier keeps a copy, and what stops a task from appearing on both sides of the split after a correction round.
Record the split with the provenance of each example so a later delivery can be checked against it.
What to report with a score
A score without its conditions cannot be reproduced. Report the model, prompt, dataset and scorer versions together with the date, and do not put two numbers in the same table if the scorer changed between runs.
Keep scoring dimensions separate: format validity, tool correctness and task success are different measurements, and averaging them hides which one failed. Where success depends on a programmatic check, the sandbox environment verifier decides the outcome.
Acceptance question
Ask for the score on a slice you can inspect, with every version named, and reproduce one item by hand. The buying guide treats that as part of a bounded pilot, and the methodology explains why this site calls such a result an observation on one task rather than evidence of general capability.
Questions & answers
How large should an evaluation set be?
Can a supplier benchmark serve as my evaluation set?
Sources for this entry
- Editorial checklists on this site — Original guidance written for this reference; the reporting fields are editorial recommendations, not a published standard.
- WebArena: a realistic web environment for autonomous agents — Public paper describing task environments, success criteria and reproducible evaluation of agent behavior; cited for the category, not as a ranked result.
Editorial checklists on this site are original guidance. Links to official product documentation help readers understand the category; they do not establish comparative performance and are not endorsements of any supplier.
Cite this entry
Best AI Agent Training Data. “Evaluation set.” Content reviewed .
https://bestaiagenttrainingdata.com/