Overview

Attest 1.1.0 “Hardened” tests AI agents with pytest and Vitest. It checks what an agent says, which tools and agents it calls, and what it changes in an external system. Record model responses once and replay them against your real code to test changes without live model calls for matching requests.

One small test

Install with uv add "attest-ai>=1.1.0" pytest, save this as test_support.py and run uv run pytest:

from attest import agent, expect


@agent("support")
def support(builder):
    builder.add_tool_call("lookup_order", args={"order_id": "O-1"}, result={"status": "delivered"})
    return {"message": "Order O-1 was delivered."}


def test_checks_the_reply_and_tool(attest):
    attest.evaluate(expect(support()).required_tools(["lookup_order"]).output_contains("O-1"))

This records sample data and makes no network calls. For your application, record real execution with an adapter or capture, then choose checks for your business rules. A recording consistent with its own trace is not proof that the behavior is correct.

Record, replay, gate

1. Capture a run and review its redacted data and drafted contracts.
2. Replay recorded model responses while the current agent code runs. Strict replay reports a mismatch instead of inventing a response; isolate tools and time-dependent state too.
3. Gate the result with policies and reviewed baselines. Failed native tests block by default; not_evaluated only warns unless you configure not_evaluated: block. Require the policy check in branch protection.

The tour shows this workflow on a synthetic double refund. The 1.1 release post explains evaluated-chain enforcement, relational effects, scheme-2 assertion ids, baseline removals and judge failure handling.

Capabilities and maturity

These tiers match the 1.1.0 engine’s capability inventory. Stable features have parity and conformance coverage; behavior changes are breaking changes. Beta features are shipped and tested but may change in a minor release. Experimental features have no stability commitment and may change or be removed in any release.

Capability	Tier	Start here
Capture, replay and merge gate	stable	Capture, replay, merge gate
Behavior and world-effect contracts	stable	Behavior rules, world fixtures and relational effects
CI policy, baselines and reports	stable	Policies, baselines, annotations, selection, CLI
Failure viewer	beta	Viewer
Mutation testing	beta	Mutation
MCP server and agent skills	beta	MCP, agent setup
LLM judges on four provider families	beta	Judges and calibration
Multi-agent trace trees	beta	Multi-agent testing
Simulation runtime	beta	Simulation, statistical verdicts
Plugin assertions	beta	Plugins
Local ONNX embeddings	experimental	Configuration
Continuous evaluation and drift detection	experimental	Continuous evaluation

Trace checking helps inspect missing evidence before testing. Verdicts explains pass, fail and not-evaluated results; SDK parity describes Python and TypeScript coverage.

Checks and providers

Free deterministic checks run before embedding and judge checks. Embedding and judge cost depends on the selected provider: OpenAI embeddings and hosted judges may cost money; local ONNX embeddings or a local judge endpoint avoid those provider API charges. ACP judges are local-only and unsandboxed; read the safety guidance before using one. Plugins define their own cost and determinism.

Layer	Assertion types	What it checks	Cost	Deterministic
L1 Schema validation	schema	Checks the output against a JSON Schema.	Free	Yes
L2 Constraint checks	constraint	Checks cost, token, latency and step-count limits.	Free	Yes
L3 Trace and behavioral checks	trace, trace_tree, effect	Checks which tools ran, in which order, with which arguments, how often per business key, across the agent tree and against the world's final state.	Free	Yes
L4 Content pattern matching	content	Checks the output text against substrings, patterns and keywords.	Free	Yes
L5 Embedding similarity	embedding	Scores semantic similarity against a reference with an embedding model: OpenAI embeddings are paid, the local ONNX model is free.	Depends on provider	No
L6 LLM-as-judge	llm_judge	Scores the output against a rubric with a judge model: OpenAI, Anthropic and Gemini judges are paid, a local ACP agent or an OpenAI-compatible local endpoint is not.	Depends on provider	No
Plugins	plugin	Custom assertions that run in the SDK process; cost and determinism are the plugin's.	Defined by the plugin	No

From attest-engine capabilities --json at engine 1.1.1.

Adapters

Adapters normalize provider responses or framework runs into traces. Availability and maturity differ by language:

Adapter	Python	TypeScript
OpenAI	stable	stable
Anthropic	stable	stable
Manual	stable	stable
Gemini	beta	beta
Ollama	beta	beta
LangChain	beta	beta
CrewAI	beta	Not available
OpenTelemetry	beta	beta
Google ADK	experimental	Not available
LlamaIndex	experimental	Not available

• beta: Shipped and tested, but the interface or behavior can still change in a minor release.
• experimental: Shipped without a stability commitment and without demonstrated demand; it can change or be removed in any release.
• stable: Covered by the parity and conformance suites; a change in behavior is a breaking change.

From attest-engine capabilities --json at engine 1.1.1.

Install and runtime

# Python
uv add "attest-ai>=1.1.0" pytest
uv run attest doctor

# TypeScript
npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"
npx attest doctor
npx vitest run

For TypeScript, run npx attest doctor before the first test: downloading the engine on a cold machine can exceed Vitest’s default timeout. The Vitest integration supports versions 3, 4 and 5; see the release notes for concurrent lifecycle limitations.

The SDK starts a Go engine subprocess and communicates over JSON-RPC/NDJSON. Published SDK packages download the matching engine version and verify its pinned hash. Source/editable installs require a local engine build and ATTEST_ENGINE_PATH. Optional local ONNX embeddings also need ONNX Runtime. See configuration for details.