System Architecture

Attest tests what an AI agent did and what it left behind. An SDK captures execution as a trace, adapters translate provider or framework data, and a local Go engine evaluates the assertions you explicitly request.

Components

Component	Responsibility
Python SDK (attest-ai)	Trace construction, expect() chains, adapters, engine lifecycle, pytest integration
TypeScript SDK (@attest-ai/core)	Trace construction, fluent assertions, adapters, engine lifecycle
TypeScript test integration (@attest-ai/vitest)	Vitest fixtures and test-outcome enforcement
Go engine (attest-engine)	Deterministic and model-based assertion evaluation over JSON-RPC
SDK plugins	Custom evaluation logic run outside the engine

These components ship as 1.1.0. Python requires 3.10 or newer. Both SDKs use the same engine protocol, but their adapter inventories differ; do not assume every Python integration is available in TypeScript.

A small test

from attest import expect

def test_refund_contract(attest, result):
    attest.evaluate(
        expect(result)
        .required_tools(["lookup_order"])
        .output_contains("refund")
        .cost_under(0.05)
    )

The result is your captured agent result. expect() builds a chain; attest.evaluate() sends it to the engine and enforces the outcome. Leaving a chain unevaluated fails the test in 1.1.0, including assertions appended after evaluation.

Design principles

Test outcomes and constraints

An agent can take different valid paths. Assert required outcomes, allowed tools, schemas, and budgets rather than making every test depend on an exact sequence. Use order assertions when the sequence itself is a business rule, such as checking eligibility before a refund.

Deterministic checks first

Layers 1–4 check schemas, numeric constraints, trace behavior, and content without calling a model. Layer 3 includes trace trees and final world-state effects. Layer 5 uses embeddings; Layer 6 uses a judge. Their cost depends on the provider: local ONNX embeddings avoid embedding API charges, while hosted models can incur charges.

The engine runs deterministic assertions before L5/L6 and short-circuits those model-based assertions after any deterministic hard failure. It does not invent new assertions or automatically choose a judge when a deterministic check passes. The model is six evaluated layers plus plugins.

See Assertion Pipeline for the full layer and scheduling model.

Missing verdicts are not passes

Results distinguish pass, soft_fail, hard_fail, and not_evaluated. Embedding and judge checks ordinarily pass at their configured threshold and otherwise hard-fail, or soft-fail when explicitly configured as soft. There is no universal 0.5/0.8 score band that decides merge permission.

Malformed judge replies, abstentions, unavailable providers, and exhausted call budgets produce explicit missing-verdict reasons. Default policy warns on not_evaluated; configure not_evaluated: block if CI must require a verdict. Soft-failure budgets and CI policy are separate from numerical scores.

Code-first contracts

Python and TypeScript tests build assertions directly in code. Trace capture and assertion evaluation are separate: recording an event does not assert anything about it. Pytest and Vitest enforce evaluated chains and report missing verdicts rather than silently treating them as successful checks.

Simulation still checks evidence

ATTEST_SIMULATION=1 runs the real engine and deterministic checks. Embedding/judge assertions return not_evaluated(simulation) unless an earlier hard failure short-circuited them. This mode does not fabricate passes, bypass the engine, or stop application code from making provider calls.

Measure cost without confusing budgets

Agent token use, cost, and latency can be recorded in trace metadata and checked alongside correctness. An absent measurement is not a measured zero. The cost of Attest’s own model-based evaluation is separate: batch usage tracks its calls, tokens, priced spend, retries, and budget refusals. Use this ledger rather than assuming assertion cost captures every billed call.

Trace data model

A trace contains trace_id, schema_version, agent_id, input, steps, output, metadata, and optional parent linkage. Each step has a type, name, arguments, result, and optional timing/agent information. An agent_call can contain a nested sub_trace.

Step type	Meaning
llm_call	Provider/model interaction
tool_call	Recorded tool interaction
retrieval	Retrieval operation
agent_call	Delegation, optionally with a nested trace

Optional started_at_ms and ended_at_ms timestamps support order and overlap checks. agent_id and agent_role support agent-specific checks. Evidence availability depends on the adapter and source events; not every integration captures a full delegation tree or real request timings.

World-effect assertions use a separate world-v1 snapshot embedded in the assertion spec. This lets tests check stored refunds or notifications, not merely whether a tool was requested. Relational sum bounds amounts by a related record; matches checks fields across records joined by a business key.

Adapter system

Provider adapters translate completed OpenAI, Anthropic, Gemini, or Ollama responses. Framework adapters consume callbacks, events, or summaries. Manual capture and OpenTelemetry conversion cover other instrumentation paths.

The released capability inventory lists 10 Python and 7 TypeScript adapters. OpenAI, Anthropic, and Manual are stable; other integrations are beta or experimental. CrewAI capture is summary-only, not a full tool/delegation trace. See Adapter System for the inventory, maturity commitments, and evidence boundaries.

Engine lifecycle and discovery

The SDK starts the engine as a subprocess and exchanges JSON-RPC 2.0 messages as NDJSON over stdin/stdout. The handshake reports the version, runtime capabilities, and limits. SDK releases require their matching ENGINE_VERSION, with version-specific binaries cached under ~/.attest/bin/<ENGINE_VERSION>/.

Python discovery checks an explicit engine path, ATTEST_ENGINE_PATH, PATH, and the versioned cache before downloading the release binary when allowed. It does not implicitly search a monorepo or local ./bin/; use ATTEST_ENGINE_PATH for a development build. A binary with the wrong version is rejected rather than silently used.

See Engine Protocol for the lifecycle and JSON-RPC reference for wire fields.

Cost tiers

Cost tiers schedule tests; they do not describe adapter maturity:

Tier	Layers	Cost
TIER_1	L1–L4, including effects and trace trees	No model API calls for evaluation
TIER_2	L5	Embedding-provider dependent
TIER_3	L6	Judge-provider dependent

uv run pytest -m attest --attest-tier=1

A tier-1 evaluation is provider-free, but the agent under test may still call a provider. Use recorded or scripted application behavior when the entire test must avoid live model calls.