Drift Detection

Drift detection asks whether a check’s score has fallen relative to its recent history. It can reveal a change in a model, inputs or tools; it does not identify the cause or establish that an agent is correct. Continuous evaluation and drift detection are experimental.

How the baseline works

The engine uses the most recent 50 passing or soft-failing scores for the same assertion id, before the result being judged. With fewer than ten eligible scores it uses the static threshold alone. Hard failures and not_evaluated results never lower the baseline.

For a judge or similarity assertion, threshold="dynamic" uses a pass bar of:

max(0.8, mean − 2·max(σ, 0.02))

The history window and constants are fixed. The direct query_drift method uses its supplied static threshold (default 0.8) as the floor instead. These are related mechanisms, not configurable window estimators.

History is stored under ATTEST_CACHE_DIR. An empty cache starts cold; stable ids and preserved history matter. Upgrading from 1.0.x to 1.1.0 changes ids and rebuilds this history.

See dynamic thresholds in the SDK judge guide for evaluated examples and fallback behavior, and the SDK continuous-evaluation guide for drift alerts. Alerting skips unevaluated rows, so missing measurements need separate monitoring.