Glossary Working definitions

Glossary

Short working definitions for the terms that decide what an agent training data supplier is actually selling, written for the people who have to write the brief.

Answer in brief

This glossary gives a working definition for the nine terms that decide what an agent training data supplier is actually delivering. Each definition names the artifact, the parts a buyer can inspect, and the question worth asking before the work is ordered.

The terms are kept deliberately apart. A demonstration is not an evaluation set, a rubric is not a verifier, and a tool-use trajectory is not a transcript of a conversation. A disappointing purchase often begins with two teams using the same word for different deliverables.

Terms used in this reference

Six of the nine terms have their own entries. Verifier, hold-out and preference data are defined here because the distinction each one carries runs through the whole buying conversation.

Term	Working definition	Where it is covered
Demonstration	An input the agent can see at inference time, paired with the action or response an expert accepted for that input.	Full entry; expert demonstrations guide
Tool-use trajectory	The ordered record of one agent run: each tool call, its arguments, what the tool returned, and where the run stopped.	Full entry; tool-use trajectories guide
Evaluation set	Held-out tasks used to measure a stated agent capability, with the scoring method agreed before collection starts.	Full entry; evaluation sets guide
Sandbox environment	A controlled task world with a starting state, a defined set of permitted actions and rules that decide success.	Full entry; sandboxes and verifiers guide
Verifier	The check that decides whether an end state meets the task objective, whether a test script, a state comparison or a scored rubric.	This index
Rubric	Explicit written criteria for judging an output or an annotation decision, with pass and fail examples for each rule.	Full entry
Provenance	The record that links an example to its source, the permission basis for its use, its versions and its review history.	Full entry
Hold-out	Data kept out of training and out of supplier tuning, then used only to measure performance.	This index
Preference data	Paired or ranked examples recording which of two candidate outputs better satisfies a stated rubric.	This index

How to use these definitions when writing a brief

A definition earns its place in a purchase only if it changes what gets delivered. Copy the definition you intend into the brief, then add the parameters that move the price: who authors the examples, which fields each record must carry, how review is handled, and which rubric version applies.

Name the artifact you are not buying as well. A supplier that delivers demonstrations is not thereby delivering an evaluation set, and one that writes a rubric is not thereby building a sandbox environment. The entity map shows how the artifacts connect; the topic guides take each through specification, review and acceptance.

Use the same vocabulary in the acceptance test. Write the sentence you will apply at the end of a pilot, for example that a sample is accepted when a named reviewer can reproduce the decision from the visible input and the recorded tool output. The buying guide turns that into a sequence, and the methodology explains what this site does and does not claim.

If a term here is missing from a proposal, ask which word the supplier uses for it; silence about evaluation sets, rubric versions or provenance is worth noticing. The questions and answers page collects the rest.

Questions & answers

Is this glossary a published standard?

No. The definitions are editorial working wording for buying conversations and are not maintained by a standards body. For a specific tool or benchmark, the official documentation or paper remains authoritative; the source register records which documents this site relies on.

Why do verifier, hold-out and preference data have no entries of their own?

Because the useful part of each definition is a distinction rather than a procedure. A verifier decides success, a hold-out set measures performance on data the model was not trained on, and preference data records a judgment between two candidate outputs. The guides on sandboxes and evaluation sets show how those distinctions are applied.

Sources for this entry

Editorial checklists on this site — Original guidance written for this reference; the working definitions are editorial, not a published standard.
Label Studio — Official description of open-source data labeling and evaluation software (software, not a ranked supplier).
Argilla — Official description of dataset review and annotation tooling for AI teams (software, not a ranked supplier).

Editorial checklists on this site are original guidance. Links to official product documentation help readers understand the category; they do not establish comparative performance and are not endorsements of any supplier.

Cite this entry

Best AI Agent Training Data. “Glossary.” Content reviewed 9 November 2026.

https://bestaiagenttrainingdata.com/