Assertion Pipeline

Assertions turn recorded agent behavior into testable contracts. Start with facts you can check directly—tool arguments, output fields, or final world state—and use embeddings or a judge only when the contract needs a semantic assessment.

Layers

Layer	Assertion types	What it checks	Cost	Deterministic
L1 Schema validation	schema	Checks the output against a JSON Schema.	Free	Yes
L2 Constraint checks	constraint	Checks cost, token, latency and step-count limits.	Free	Yes
L3 Trace and behavioral checks	trace, trace_tree, effect	Checks which tools ran, in which order, with which arguments, how often per business key, across the agent tree and against the world's final state.	Free	Yes
L4 Content pattern matching	content	Checks the output text against substrings, patterns and keywords.	Free	Yes
L5 Embedding similarity	embedding	Scores semantic similarity against a reference with an embedding model: OpenAI embeddings are paid, the local ONNX model is free.	Depends on provider	No
L6 LLM-as-judge	llm_judge	Scores the output against a rubric with a judge model: OpenAI, Anthropic and Gemini judges are paid, a local ACP agent or an OpenAI-compatible local endpoint is not.	Depends on provider	No
Plugins	plugin	Custom assertions that run in the SDK process; cost and determinism are the plugin's.	Defined by the plugin	No

From attest-engine capabilities --json at engine 1.1.1.

In 1.1.0, there are six evaluated layers plus plugins. trace, trace_tree, and effect all run in Layer 3. Simulation is a runtime mode, not another layer.

Evaluation order

The engine stably groups assertions by layer rather than preserving request order:

1. L1–L4 run sequentially, including trace-tree and world-effect checks in L3.
2. SDK-computed plugin results are reported. They neither depend on nor gate L5/L6.
3. L5/L6 run concurrently with a bounded worker pool, default 8, configurable with ATTEST_EVAL_CONCURRENCY.

Any L1–L4 hard failure prevents L5/L6 calls in that batch. Every affected assertion still receives a not_evaluated row with reason short_circuited. This is unconditional, not an optional fail_fast flag. In simulation, otherwise eligible L5/L6 assertions receive not_evaluated(simulation). Without a registered provider they receive not_evaluated(provider_not_configured).

Match response rows by assertion ID. Do not rely on their position or treat batch total_duration_ms (summed assertion durations) as wall-clock latency. There are no fixed latency or per-assertion price guarantees.

Build and evaluate a chain

from attest import expect

def test_refund_contract(attest, result):
    chain = (
        expect(result)
        .required_tools(["lookup_order"])
        .output_contains("refund")
        .cost_under(0.05)
    )
    attest.evaluate(chain)

expect() builds a chain; it does not execute it. Pytest and Vitest fail a test that leaves a chain unevaluated, including assertions added after an earlier evaluation. This also applies to chains made in fixtures and concurrent Vitest tests.

L1: Schema validation

Schema checks validate a trace target against JSON Schema. For example:

attest.evaluate(expect(result).output_matches_schema({
    "type": "object",
    "required": ["status"],
    "properties": {"status": {"type": "string"}},
}))

Use tool_args_match_schema() for tool argument shapes. Invalid schemas and schema violations fail the assertion. Schema references cannot load files or fetch network resources; references within the schema document can resolve.

L2: Constraints

Constraints compare recorded values, such as agent cost, token use, latency, and step counts, against bounds:

attest.evaluate(
    expect(result).cost_under(0.05).tokens_under(3000).latency_under(2000)
)

Latency is in milliseconds. Agent cost assertions inspect the trace’s metadata.cost_usd; they do not limit Attest’s embedding or judge spend. Missing evidence must not be interpreted as a recorded zero.

L3: Trace and behavioral checks

Tool checks inspect what ran, in what order, with which arguments, and how often per business key. Trace-tree checks inspect delegation, agent output, and relationships between nested traces. Temporal checks require recorded timestamps; an adapter cannot recover timing or identity the source did not provide.

attest.evaluate(
    expect(result)
    .required_tools(["lookup_order"])
    .forbidden_tools(["delete_account"])
    .tools_called_in_order(["check_eligibility", "process_refund"])
)

TraceTree also supports direct inspection of nested agent_call steps and their sub_trace values. These are deterministic checks, not judge calls.

Final world state and relational effects

An effect assertion checks a supplied world-v1 snapshot, not the trace’s tool steps. A tool call saying “refund” does not prove that only one refund was stored. count, unique, and idempotent check record counts and keys; sum and matches compare amounts or related records.

from attest import expect, related

snapshot = world.state()  # a running world helper's snapshot
attest.evaluate(expect(result).world(snapshot).sum(
    "refunds", "amount_cents",
    at_most=related("orders", "amount_cents", on="order_id"),
))
attest.evaluate(expect(result).world(snapshot).matches(
    "notifications", "customer_id",
    related("orders", "customer_id", on="order_id"),
))

• A numeric sum(..., at_most=100) bounds the total of matching records. A related bound groups source records by on and checks each sum against the related record’s field.
• matches requires each matching source record’s field to equal the related record’s field, joined using the same on field on both resources. It is record equality, not regex matching.
• where and keyword filters are exact-match source-record filters. They do not filter the related resource.
• Sums use exact decimal arithmetic. Strings, nulls, booleans, and missing numeric fields are not silently counted as zero.
• Missing join keys or related records fail the contract; duplicate related keys are ambiguous evidence and fail. Mismatched or missing comparison fields fail matches.
• Empty source selections do not prove a record exists: combine these checks with count when existence is required.
• A null snapshot yields not_evaluated(missing_evidence). A missing or malformed snapshot is an invalid effect spec and fails.

Both Python and TypeScript provide related and world-effect checks; Python uses snake_case and TypeScript camelCase option names where applicable.

L4: Content

Content checks compare a selected trace target to substrings, keywords, or Go regular expressions. SDK output helpers select output.message; the engine is not limited to that field.

attest.evaluate(
    expect(result)
    .output_contains("refund")
    .output_not_contains("internal_api_key")
    .output_matches_regex(r"ORD-\d+")
)

Substring and keyword checks default to case-insensitive comparisons. Regex matching uses the pattern itself; use regex flags when needed. It is deterministic, not “near deterministic.” A regex can check a chosen PII pattern but is not a general guarantee that output is free of sensitive data.

L5: Embedding similarity

Embedding checks compare output and reference vectors using cosine similarity:

attest.evaluate(expect(result).output_similar_to(
    "The order is outside the return window.", threshold=0.7,
))

ATTEST_EMBEDDING_PROVIDER accepts auto, openai, or onnx. OpenAI calls can incur cost; local ONNX embeddings do not incur provider API charges. Cache hits can reuse text embeddings. Provider, model, input, and cache state affect cost and runtime; there is no fixed dollar price per assertion.

L6: LLM judge

A judge scores the selected output against criteria and a rubric:

attest.evaluate(expect(result).passes_judge(
    "The response explains why the refund was denied.", threshold=0.8,
))

The Python defaults are rubric="default", threshold=0.8, and soft=False. Supported judge providers are openai, anthropic, gemini, and local-only acp. Anthropic and Gemini require ATTEST_JUDGE_MODEL. A custom endpoint uses ATTEST_JUDGE_BASE_URL and requires ATTEST_JUDGE_API_KEY; exported standard provider keys are not sent to it. ACP is refused in CI and under a USD budget.

Fail-closed replies and caching

A malformed judge reply, explicit abstention, or credential rejection is not a passing score. Results distinguish judge_malformed, judge_abstained, and provider_auth; other unavailable-provider failures use provider_unavailable. Authentication rejections (401/403) are not retried. Only transient HTTP failures (408, 429, 5xx) and qualifying network errors are retried, subject to budgets and deadlines.

Judge cache identity includes criteria and target content, rubric version, repeat count and bias probes, model, and provider/endpoint scope. Changing the threshold reclassifies a cached score against the current threshold; it does not reuse the old pass/fail decision. Invalid specs are rejected before cache lookup. A cache hit incurs no new judge call. Abstentions and malformed replies are not cached as scored verdicts. Repeated judge evaluation uses the median of scored samples and exposes sample/variance metadata; it is not the same as repeated independent tests.

Scoring is assertion-specific

For ordinary embedding and judge assertions, meeting the configured threshold passes. A score below it is hard_fail by default, or soft_fail with soft=True. There is no universal rule that all scores between 0.5 and 0.8 are soft failures. Neither a soft failure nor not_evaluated automatically grants merge permission: CI policy decides that.

Dynamic thresholds (threshold="dynamic") use historical passing and soft-failing scores. The estimator uses up to 50 eligible prior runs and needs 10 to activate. Its bar is max(static_threshold, mean - 2 * max(stddev, 0.02)); a previously passing score below the active bar becomes hard_fail, or soft_fail when the assertion specifies soft=True. Existing failures keep their status. If the history store cannot be read, threshold_source: dynamic_unavailable exposes the static fallback.

Plugins

Plugins execute custom logic in the SDK process. The SDK submits a computed result using submit_plugin_result before evaluating its plugin assertion. The engine retrieves the stored result by trace ID, assertion ID, and plugin name; it does not wait for a later submission. Plugin cost and determinism depend on the plugin, not its layer number.

Budgets, simulation, and policy

ATTEST_MAX_SOFT_FAILURES limits soft failures per engine session. Exceeding it sets the batch’s budget_exceeded flag without dropping rows. ATTEST_MAX_JUDGE_CALLS and ATTEST_BUDGET_USD, plus optional per-batch limits, cap Attest-initiated model calls. Refused model work returns not_evaluated(budget_exhausted). Use batch usage for model-spend accounting, not only assertion cost or total_cost; embedding result costs do not provide that full ledger.

ATTEST_SIMULATION=1 runs deterministic checks for real and disables L5/L6 evaluation. It neither bypasses the engine nor fakes passes, and it does not prevent the application under test from making model calls. Under the default pytest/Vitest enforcement, a missing verdict skips the test unless a failure is also present; pytest’s --attest-strict makes missing verdicts fail. The default CI policy warns on not_evaluated; set not_evaluated: block to require complete verdicts.

Cost tiers

Tier	Layers	Use
TIER_1	L1–L4	Deterministic development checks, including effects and trace trees
TIER_2	L5	Embedding checks; provider-dependent cost
TIER_3	L6	Judge checks; provider-dependent cost

uv run pytest -m attest --attest-tier=1

See the JSON-RPC reference for wire fields and the Adapter System for how traces acquire evidence.