Migrating from Manual Testing

Keep human review for judgment calls, and automate the checks you can state precisely. Attest 1.1.0 lets you test a reply, a tool sequence and an external effect inside pytest or Vitest. Passing tests cover the rules you wrote, not every possible defect.

Start with one checklist item

Suppose the rule is: “Look up the order and mention its id in the reply.”

uv add "attest-ai>=1.1.0" pytest

Save this complete, no-network example as test_support.py:

from attest import agent, expect


@agent("support")
def support(builder):
    builder.add_tool_call("lookup_order", args={"order_id": "O-1"}, result={"status": "delivered"})
    return {"message": "Order O-1 was delivered."}


def test_order_lookup(attest):
    attest.evaluate(expect(support()).required_tools(["lookup_order"]).output_contains("O-1"))

Run uv run pytest. For your application, instrument the real agent with a framework adapter or record calls explicitly on the builder. The decorator alone does not intercept every operation.

Translate the rest deliberately

Checklist item	Check	Evidence needed
Required fields and types	JSON Schema assertion	Structured output
Required or forbidden text	Content assertion	Reply text
Response time or spend limit	Constraint assertion	Actual timing and usage metadata
Tool order or at-most-once per order	Behavior contract	Tool names, arguments and ordering
Refund total or notification recipient	World-effect contract	Isolated final state and related records
Helpful, clear or other subjective quality	Calibrated judge	A rubric and the evidence needed to score it

Choose latency and spend limits from your application’s requirements. A text check cannot prove a payment happened exactly once: test the ledger too. The tour illustrates that distinction.

Make failures visible

Evaluate every assertion chain, including after adding more assertions. Use capture and replay to preserve an incident as a reviewed regression fixture; control external tools and test isolation as well as model responses.

A missing provider or malformed judge reply is not_evaluated, not a pass. The policy defaults to warning on that result; use not_evaluated: block for required measurements and require the gate in branch protection. Review verdicts before aggregating repeated trials.

Continue with Python setup, TypeScript setup, behavior contracts and world fixtures.