Glossary Rubric

Rubric

Explicit criteria for judging an output or an annotation decision, with pass and fail examples that let two reviewers reach the same verdict.

Answer in brief

A rubric is the written test a reviewer applies to an output or to an annotation decision. It states the criteria, the order in which they are applied, and what counts as a pass or a fail on each one.

The part buyers underestimate is calibration. A rubric that reads clearly but leaves boundary cases to judgment will produce disagreement, and a disagreement recorded as ground truth is difficult to remove from a delivered set.

Definition

A rubric is an explicit set of criteria for judging outputs or annotation decisions. Each criterion should be decidable by a reviewer who was not present when it was written, which means pass and fail examples for every rule and a stated order for when rules conflict.

A rubric is not a score sheet with weights alone. Weights say how much a dimension matters; rules and examples say what to decide. The evaluation set entry covers how scored items are held out.

One rule, one pass and one fail example

For every rule, include a worked pass and a worked fail drawn from the real task, taken at the boundary a reviewer will meet. A fail example that is obviously wrong teaches nothing; the useful fail is the plausible answer that is still unacceptable.

Name the boundary cases rather than leaving them to be discovered: a correct response that arrives late, an escalation that was right for the wrong reason, a tool call that succeeded while passing a forbidden argument.

Resolving disagreement

Put two reviewers on the same sample before collection scales. Disagreement is either settled by the rubric or it exposes a gap; resolve the gap by revising the rule in writing, not by an unwritten ruling the next reviewer will never see.

Ask what happens when reviewers still disagree after a revision: who decides, and how the decision is recorded. The sandbox environment entry covers the same question where the judge is a program.

Rubric version in the dataset

Rubrics change during collection, which is normal. Record which version governed each example so an item accepted under an earlier rule is not silently re-graded against a later one, leaving two identical examples judged by different standards.

When a rule changes, decide whether earlier items are re-reviewed or kept under their original version, and keep that decision in the provenance record.

Acceptance question

Ask for the rubric and the result of a calibration run on a small set: two reviewers, independent decisions, then a written resolution log. Accept when the log shows disputes settled by rules you can read, not by a manager you cannot see. The buying guide asks whether rubric design is inside the scope of the engagement.

Questions & answers

Should a rubric award partial credit?

Only where the grader can apply the same partial rule twice and get the same answer. Partial credit is useful for diagnosis; if it affects acceptance, write the weights into the rubric and check them on examples the buyer has already read.

Who should write the rubric?

The buyer owns the definition of success; the supplier can draft the wording. Write the objective first, then ask for the rules the supplier would apply. If the same supplier writes the rubric and grades against it, review the rubric before delivery.

Sources for this entry

Editorial checklists on this site — Original guidance written for this reference; the calibration approach is editorial, not a published standard.
Label Studio — Official description of open-source data labeling and evaluation software (software, not a ranked supplier).

Editorial checklists on this site are original guidance. Links to official product documentation help readers understand the category; they do not establish comparative performance and are not endorsements of any supplier.

Cite this entry

Best AI Agent Training Data. “Rubric.” Content reviewed 9 November 2026.

https://bestaiagenttrainingdata.com/