How to buy AI agent training data

Step 1 — Describe the task before you describe the data

A brief that opens with a file format is written backwards: start with the decision the agent must make and the state it sees at inference.

• Task boundary. What the agent must do, what is out of scope, and where a run stops and escalates.
• Intended behavior. The action an expert accepts, with acceptable alternatives and prohibited shortcuts.
• Success definition. What a reviewer can see in the artifact that shows the run succeeded.
• Inputs and tools. The state visible at inference, the tools and APIs allowed, and anything kept out of the prompt.
• Output schema. The fields you expect, and how partial runs, retries and failures are represented.
• Constraints. Prohibited actions, data that may not be retained, reviewer eligibility, and team obligations.

Finish with one approved sample task. The glossary settles vocabulary; the topic guides cover demonstrations, trajectories, evaluation sets and sandboxes and verifiers.

Step 2 — Compare proposals on evidence, not on presentation

Ask every supplier the same questions in writing, then compare the answers, not the decks.

• Who supplies the experts. Recruitment, domain experience, and how eligibility is checked.
• Disagreement. Whether a second reviewer sees disputes, who decides, where it is logged, and when the rubric changes.
• Delivered artifacts. Formats, schemas, reviewer notes, and how failed or partial runs appear.
• Corrections. The process when your reviewers reject part of a delivery, and who pays for rework.
• Buyer responsibilities. Accounts, tool and environment access, reviewer time, and a named contact.
• Certification claims. The issuing body, the scope and the date. A badge with no named issuer is decoration.

Ask for two worked examples on your approved sample, not a catalog extract. A file nobody can review is not the cheaper purchase, and a fluent answer that calls the wrong tool is still a failed example.

Step 3 — Buy through a fixed, bounded pilot

A pilot is a task test, not a file-format demonstration. Fix five things before work starts: the task, the deliverables, the acceptance method, the scorer, and the go/no-go date.

• Task. The approved sample or a similar small set, including one ambiguous and one exceptional case.
• Deliverables. Named files, schema, reviewer notes, and how failures appear.
• Acceptance. The rubric or verifier, the pass threshold, and the share your reviewers inspect.
• Commercials. A fixed price or a ceiling you set, with rework terms.
• Decision. The meeting date and the evidence required before a second order.

Expand to volume only after a documented result; failing before a volume commitment is cheaper. The bounded pilot guide covers case selection and thresholds, and you can request a shortlist of candidates to run it with.

Questions to paste into a brief

Copy these into the request and require written answers from every supplier.

1. Which examples are collected on our task, and which come from a general catalog?
2. Who authors each example, and what domain experience can you evidence?
3. What does the agent see at inference, and what is deliberately excluded?
4. How are tool calls, arguments, observations and stop reasons recorded?
5. What happens when reviewers disagree, and where is the decision logged?
6. Which files are delivered, in which format, and how are failed runs represented?
7. What is your correction policy if our reviewers reject part of a delivery?
8. Which evaluation items are held out, and how is contamination prevented?
9. What do you need from us: accounts, tools, environment access or reviewer time?
10. Is the scope data only, data with rubric design, or environment design?

Five proposal red flags

One is a question to resolve before signing; two is a reason to look elsewhere.

1. A quote on volume and price before anyone discussed the task boundary.
2. A certification claim with no named issuing body, scope or date.
3. “Human in the loop” with no reviewer profile and no escalation path.
4. Deliverables named only by file format, with no acceptance method.
5. A multi-month volume commitment requested before a pilot result.

Questions & answers