1. Decision and task unit
State the decision the work supports. Define whether a reviewer assesses a response, conversation, pull request, document, image, or workflow run.
The brief worksheet
2. Context and reviewers
Specify the product surface, intended user, language, domain, constraints, and practical experience needed to judge the work. Separate essentials from trainable preferences.
3. Shareable materials
Use only approved instructions and a sanitized sample. Name what cannot be shared during recruitment or onboarding.
4. Observable acceptance
Bring a rubric, examples, an uncertainty path, pilot scope, review cadence, and a named decision-maker for changes.
Define the work before the role
“AI evaluator” can mean preference ranking, factuality review, code assessment, red-teaming, data-quality checks, or a domain-specific rubric. Name the task, expected language and domain, and the product context rather than relying on a broad title.
For coding work, name the languages, frameworks, test approach, and defects to assess. For specialist work, have the domain owner review the brief. “Expert” alone is not an acceptance criterion.
Remote work does not automatically mean worldwide eligibility. State real location or work-authorisation limits, and share only material approved for the process.
Bring a small evaluation package
Task and rubric
Describe what the evaluator should and should not decide. Use observable criteria with acceptable, borderline, and unacceptable examples.
Calibration and uncertainty
Use a small calibration set with reference judgments and rationales. Record uncertainty, abstentions, or missing context.
Disagreement handling
Decide how reviewers surface disagreement, who resolves it, and how rubric changes are recorded and shared.
Acceptance conversation
Ask what evidence demonstrates fit, which requirements can be taught, how the pilot is reviewed, and who owns final acceptance.
Use a pilot to learn, not to manufacture a score
Compare reviewers on the same small sample. Differences may show ambiguous criteria, missing context, different domain knowledge, or an interface problem. Record the rubric version and what changed instead of reducing the pilot to one headline number.
NIST’s voluntary AI Risk Management Framework is a useful reference for structured questions about context, evaluation, and escalation. This guide does not imply NIST conformance. Read the NIST AI RMF.
Discuss a hiring brief