Judge Calibration
A model judge is a measurement, not a correctness oracle. Calibration compares its scores with human labels for the same rubric, provider and model. The judge subsystem is beta.
Use deterministic assertions for concrete requirements first. Use a judge for subjective criteria, then inspect agreement, repeated-sample variance and bias probes before making its scores a merge requirement.
Keep measurements comparable
Section titled “Keep measurements comparable”- Results identify the provider, model, rubric version and prompt hash.
- Repeated samples use the median of scored replies; abstentions and malformed replies are not fabricated scores.
- Calibration labels are scoped to provider and model as well as rubric, rubric version and prompt hash. Do not pool unrelated judges.
- Repeats and bias probes make additional judge calls; configure call and USD budgets deliberately.
In 1.1.0 a malformed reply is not_evaluated(judge_malformed) and rejected credentials are not_evaluated(provider_auth), not low scores. Abstention also yields not_evaluated. The default policy only warns on these results; set not_evaluated: block when a required measurement must block the merge.
The SDK judge guide is the source for provider setup, local ACP safety, dynamic thresholds, budgets and per-model calibration. The verdict guide explains test enforcement; policies explains merge enforcement.