Context card
From behavior objective to evaluation criterion
How does a written behavior rule become a score in a model evaluation?
The short answer
A behavior rule becomes measurable when it is broken down into named sub-behaviors, each with test prompts and a rubric that defines which response counts as aligned. The resulting score reports conformity of sampled outputs to that rubric, for the version and prompt set tested. It does not show that the rule is right or that the assumptions behind the rule are true.
For: Engineers and technical writers documenting AI evaluations
Key points
- Policy sentences are not testable until they are decomposed into observable sub-behaviors.
- A rubric turns a value statement into a verifiable requirement.
- A pass rate is bound to one model version, one prompt set and one rubric version.
- Conformity to a normative rule is not empirical proof of the rule's premises.
- Evaluation reports should label which results measure safety, accuracy or policy adherence.
The context
The chain from sentence to score
A sentence in a behavior policy is not testable as written. Evaluation work turns it into a hierarchy: a named behavior such as transparency about capabilities, sub-behaviors that each describe one observable aspect, prompts that create the situation, and a rubric stating which answer counts as aligned. At that point the sentence has become a requirement in the engineering sense — specific enough to be checked, and traceable back to its source. The scoring itself can be done by human raters, by rule-based checks or by a model used as judge, and each option brings its own error profile and its own documentation duty.
What a passing score actually shows
An evaluation result shows how sampled outputs of one model version relate to one rubric, under one prompt set. That is a genuine finding and narrower than it looks. It does not show that the rule is a good rule, that the premises behind it hold, or that behavior stays stable on inputs outside the set. A score against a self-description rule, for example, measures whether the model answers the way policy demands; it settles nothing about the inner states the policy asserts or denies.
Keeping kinds of results apart
Verification vocabulary helps here: evidence is always evidence for a stated claim, and the claim has to be written down before numbers mean anything. Useful evaluation documentation therefore labels each result by what it measures — refusal in safety-relevant situations, factual accuracy, or adherence to a chosen institutional policy — and names the version of both the policy and the prompt set. Readers can then see which figures would move if the policy changed and which would not.
Without that labeling, a dashboard of pass rates invites a false reading: that normative choices have been empirically confirmed. The honest framing is narrower and more useful. The organization decided what it wants, wrote it down, and measured how often its models do it — three steps that deserve three separate statements in any documentation that cites the numbers.
Questions readers ask next
- Who decides what counts as an aligned response?
- The rubric author, in line with the policy. That is why rubrics, their versions and their examples belong in the published evaluation material, not only the aggregated score.
- Can a model be used to grade another model?
- It is common for large prompt sets. It needs its own validation against human judgments, and the grader model and its version should be documented with the result.
- How do regulated projects use such scores?
- As one input among others in testing and risk documentation. A score supports a specific claim about tested behavior; it does not by itself establish that a legal requirement has been met.
Sources
- Introducing Model Spec Evals — OpenAI, October 6, 2026
- Model Spec — OpenAI, December 18, 2025
Review log and changes
Every context card is checked against its sources before it is published, and again whenever it changes; the date under the byline is the last review. Corrections (something was wrong) and additions (something was missing) are logged below with date and time (Berlin time). Typos, formatting and link fixes are not listed.
Reviewed
No corrections or additions since publication.