Continuous Evaluation

Continuous evaluation runs checks on sampled application traces in the background. Use it to send a webhook or Slack alert after a hard failure or a score drop. This feature is experimental, not a substitute for a required CI gate.

The Python and TypeScript ContinuousEvalRunner use the same assertion DSL as tests. Give the runner a stable, required name: it scopes assertion identities and therefore their history. Sampling controls how many traces are evaluated, not how many application requests are served.

What alerts mean

• A hard_fail alerts immediately, including during cold start.
• A pass or soft_fail alerts only when drift is detected.
• A not_evaluated row is not queried for drift and does not alert. Monitor provider and runner errors separately.

Drift needs at least ten passing or soft-failing history rows; hard failures do not lower the baseline. In 1.1.0 assertion identities changed, so history from older ids starts cold. Judge and embedding checks require their configured provider; their cost depends on that provider.

Use the SDK continuous-evaluation guide for complete Python and TypeScript setup, queue bounds, error handling, alert payloads and shutdown behavior. Python start(), submit() and stop() are awaited. TypeScript start() and submit() are synchronous, but stop() must be awaited to drain work.

See also drift detection and configuration.