Engine JSON-RPC Protocol

The engine is a local Go subprocess that checks recorded agent behavior. The SDK handles process startup and JSON messages, so most tests only need to build a trace and evaluate an assertion chain.

Lifecycle

1. The SDK starts the engine and sends initialize with protocol version 1.
2. The engine reports its version, runtime capabilities, compatibility, and limits.
3. The SDK sends evaluate_batch requests containing a trace and assertions.
4. The SDK sends shutdown; the engine returns session statistics.

For example, a test uses the SDK rather than sending protocol messages directly:

from attest import expect

def test_refund_message(attest, result):
    chain = expect(result).output_contains("refund")
    attest.evaluate(chain)

An expect() call only builds assertions. In 1.1.0, pytest and Vitest fail tests with unevaluated chains, including chains with assertions added after a previous evaluation.

Transport

Messages use JSON-RPC 2.0 over UTF-8 newline-delimited JSON on stdin/stdout. Each wire message occupies one line; stderr is reserved for logs. A request contains jsonrpc, id, method, and params. A response echoes the ID and contains either result or error.

The stdio engine dispatches requests sequentially (max_concurrent_requests: 1). That is separate from the concurrency of embedding/judge assertions within a batch, whose default worker limit is 8. SDKs route responses by request ID, not arrival order, and reject mismatched assertion IDs. Request timeouts restart the engine and fail pending requests rather than letting late responses reach another caller.

In 1.1.0 string, numeric, and null request IDs round-trip unchanged. A valid request without an ID is a notification and gets no reply. The engine does not emit unsolicited drift notifications; drift diagnostics travel inside responses.

Method responsibilities

Method	Responsibility
initialize	Negotiate protocol version, capabilities, and limits once per session
evaluate_batch	Evaluate a trace’s assertions with optional batch budget and test context
submit_plugin_result	Store an SDK-computed plugin result before its batch evaluation
validate_trace_tree	Validate and summarize a nested trace tree
query_drift	Read the newest recorded score against a historical baseline
generate_user_message	Generate a simulated user’s message using a configured judge provider
shutdown	Return session counters and enter shutdown

generate_user_message is registered only when a judge provider is available. It can call a model; do not confuse it with provider-free assertion simulation.

Evaluation results

Each requested assertion gets a result row. The engine groups work by layer, so match rows by assertion_id, not by input position. Results distinguish pass, soft_fail, hard_fail, and not_evaluated. The last means no verdict was produced, with a reason such as simulation, provider_auth, judge_malformed, or budget_exhausted.

Deterministic checks run first. A deterministic hard failure short-circuits embedding and judge work, returning explicit not_evaluated(short_circuited) rows. Plugin results are already computed in the SDK: reporting them neither depends on nor gates embedding/judge execution.

ATTEST_SIMULATION=1 keeps the engine and deterministic evaluation active; it disables L5/L6 verdicts rather than manufacturing passes. It does not stop the agent under test from making network calls.

Costs and budgets

The agent’s trace cost (metadata.cost_usd) is distinct from the cost of Attest’s evaluation. Batch usage records Attest-initiated model calls, tokens, priced USD spend, retry spend, and budget refusals. Assertion cost and batch total_cost are not a replacement for that ledger; unpriced calls also make USD totals incomplete.

Batch budget.max_judge_calls and budget.max_cost_usd apply in addition to session limits. Exceeding the soft-failure session budget sets budget_exceeded without discarding assertion rows. Default policy only warns on not_evaluated; configure not_evaluated: block when an absent verdict must block CI.

Limits and full reference

The handshake advertises a 10 MiB trace limit and 10,000 steps per trace. Capabilities depend on provider configuration and simulation; do not assume one fixed capability list.

See the JSON-RPC reference for method parameters, result fields, drift estimation, and error codes, and the Assertion Pipeline for evaluation and scoring rules.