A buying reference for agent training data

Content reviewed

Best AI Agent Training Data Procurement Reference

Glossary Demonstration

Demonstration

An input the agent can see at inference time, paired with the action an expert accepted for it.

Answer in brief

A demonstration is one input the agent can see at inference time, paired with the action or response a domain expert accepted for that input. The pair is the unit of value: an answer without its input teaches little, and an input without an accepted outcome teaches nothing.

Sets are usually bought as a mix of routine, ambiguous, rare and escalation cases. Whether a pair is usable depends on whether a reviewer would sign it again today, not on how many pairs there are.

Definition

A demonstration is an input paired with an accepted example of the desired agent behavior. Accepted means that a named reviewer with standing in the domain approved that action for that input, under a rubric that existed at the time of review.

The definition has a boundary. If the example depends on context the deployed agent will not have — a private note, a verbal instruction, a database state that will not exist at runtime — it is not a demonstration of the deployed behavior. The guide to expert demonstrations develops this visibility test.

What a usable demonstration must contain

A usable pair can be reviewed without asking the supplier to explain it:

  • The input as the agent receives it, including tool output the agent may see.
  • The accepted action, in the output format the agent must produce.
  • The tools the example uses, and any tool it must not call.
  • The reviewer who accepted it and the rubric version that governed the decision.
  • At least one exception case: a refusal, an escalation, or a correct early stop.

Review the decision, not the prose

A fluent answer that selects the wrong tool is still a failed example. Review the decision behind the pair — which tool the accepted action implies, what state the agent was in, why the run stopped — rather than the prose.

Calibration matters more than throughput. Put two reviewers on the same sample and resolve disagreements against the rubric rather than by seniority. Where the rubric cannot settle a disagreement, revise the rubric before collection continues.

Acceptance question

Ask the supplier to run two of your own inputs and show the resulting pairs. Accept the sample only when the visibility matches your agent's runtime and the action is one your own expert would sign — the check the buying guide places before price, and the pilot guidance turns into a decision.

Questions & answers

Does every demonstration need a written rationale?
No, and a rationale is not the product. A short note helps where the decision is contestable. What every pair must carry is the visible input, the accepted action and the rubric version; read any narrative for context the agent will never see.
How is a demonstration different from an evaluation set?
Purpose. A demonstration teaches an accepted behavior and may be trained on; an evaluation set measures behavior on tasks kept out of training. If the same input appears in both, ask how the hold-out was protected.

Sources for this entry

  1. Editorial checklists on this site — Original guidance written for this reference; the definition and review questions are editorial, not a published standard.
  2. Label Studio — Official description of open-source data labeling and evaluation software (software, not a ranked supplier).

Editorial checklists on this site are original guidance. Links to official product documentation help readers understand the category; they do not establish comparative performance and are not endorsements of any supplier.

Cite this entry

Best AI Agent Training Data. “Demonstration.” Content reviewed .

https://bestaiagenttrainingdata.com/