Designing a bounded pilot

Fix the scope in four lines

Write the scope as a single page: one task family, one deliverable, one acceptance method and one budget range you control. Name the people on both sides who can accept or reject the delivery. Anything not on the page is out of scope, including extra formats, extra case types and unlimited revision rounds.

Keep the volume small enough that you can review every item. A pilot that produces more than you can inspect tests your review capacity, not the supplier’s work. If the deliverable is a sample of expert demonstrations or evaluation items, ask for the schema and one worked example before work starts.

The buying guide covers the brief in more detail; this page is about the commitment. Do not expand scope mid-pilot. If the task turns out to be wrong, stop and rewrite it rather than accepting a moving target.

Choose routine, ambiguous, rare and escalation cases

A pilot made only of easy examples is a formatting demonstration. Cover four case types: routine cases that a competent person handles quickly, ambiguous cases where the correct behavior has to be agreed, rare cases that need a documented policy, and escalation cases where the right action is to stop and hand off.

Each case type needs a rule before collection: what the accepted behavior looks like, who decides when two reviewers disagree, and how the decision is recorded. A supplier that cannot describe its disagreement process will produce a set whose labels you cannot audit.

Use inputs you can share safely. If your own logs cannot leave the organization, give a written description of the task and let the supplier collect examples, then check the first batch against your description before the rest is produced.

Agree outputs and thresholds before starting

Define the artifact precisely: file format, field names, how accepted, partial and rejected items are represented, and what reviewer notes accompany each item. Then define the acceptance test in terms you can check yourself — for example, every item parses against the schema, each slice has the agreed count, and a random sample meets the written rubric.

Write the decision threshold before the results arrive. A workable form is three outcomes: pass and expand, revise and re-run a smaller round, or stop. State what evidence each outcome requires. Deciding the threshold after seeing the delivery turns the pilot into a negotiation.

Reserve time to read the rejected items and the reviewer notes. A delivery that shows only accepted work is a portfolio, not a result.

What the pilot decides, and what expansion depends on

The pilot answers one question: does this supplier produce work that passes our acceptance method on our task at this volume? Expansion should depend on the same method holding at a larger volume and on a cost model you have already seen. Ask for the cost model — per item, per hour, per environment run — before the pilot, so the second quote is not a surprise.

Record the decision in a short note: what was delivered, how it scored against the threshold, which slices failed, and what happens next. The note is what makes the next conversation about scope rather than impressions.

If an introduction through this site leads to a paid engagement, the site may receive a referral fee. That does not change the pilot design, the acceptance method or your decision; the about page explains the disclosure.

Pilot mistakes to avoid

• Treating this website as proof that a pilot already succeeded. It is editorial guidance, not a report of results.
• Skipping the approved sample. Without one, you are measuring the supplier’s demo against your description of the task.
• Accepting a file-format demonstration as task success. A well-formed export of wrong decisions is still wrong data.
• Leaving the reviewer and the disagreement process unnamed, then disputing labels after delivery.
• Changing the acceptance criteria after seeing the first batch, which makes the result impossible to interpret.
• Expanding before the expansion criteria are met, because the first batch looked good.

Acceptance checklist

• One page with task, deliverable, acceptance method, budget and a named decision owner.
• Case mix covers routine, ambiguous, rare and escalation cases.
• An approved sample or written task description agreed before work starts.
• Artifact format and schema fixed, including rejected and partial items.
• Thresholds and the go or no-go rule written before results are delivered.
• A random sample and the rejected items reviewed by the buyer.
• Reviewer notes and the disagreement process documented.
• Expansion criteria and cost model agreed before the second quote.

Questions & answers

Related reading

• Topics in this reference
• Evaluation sets for agents
• Sandboxed environments and verifiers
• How to buy AI agent training data
• Request a provider shortlist