Python Setup

Installation

Install the attest-ai package and pytest:

uv add "attest-ai>=1.1.0" pytest

For framework-specific adapters, install extras:

uv add "attest-ai[openai]>=1.1.0"       # OpenAI adapter (stable)
uv add "attest-ai[langchain]>=1.1.0"    # LangChain support (beta)
uv add "attest-ai[llamaindex]>=1.1.0"   # LlamaIndex support (experimental)
uv add "attest-ai[google-adk]>=1.1.0"   # Google ADK support (experimental)
uv add "attest-ai[otel]>=1.1.0"         # OpenTelemetry support (beta)

Create an Agent

The @agent decorator supplies a TraceBuilder and returns an AgentResult. Your code or adapter must record the LLM and tool calls; the decorator does not intercept arbitrary calls automatically. This is the quickstart example from attest-examples:

quickstart/agent.py

"""Simple customer support agent for Attest quickstart example."""

from __future__ import annotations

from typing import Any

from attest.agent import agent
from attest.trace import TraceBuilder


@agent("customer-support")
def customer_support(builder: TraceBuilder, order_id: str) -> dict[str, Any]:
    """Customer support agent that processes refund requests.

    Args:
        builder: TraceBuilder for constructing the trace
        order_id: The order ID to look up and potentially refund

    Returns:
        A dictionary with the refund result
    """
    # Simulate LLM call to understand the customer's request
    builder.add_llm_call(
        name="gpt-4",
        args={
            "messages": [
                {
                    "role": "user",
                    "content": f"Customer wants to return order {order_id}. Process the refund.",
                }
            ],
        },
        result={"response": "I will process the refund for this order."},
    )

    # First tool: look up the order details
    builder.add_tool_call(
        name="lookup_order",
        args={"order_id": order_id},
        result={
            "order_id": order_id,
            "status": "delivered",
            "amount": 89.99,
            "customer": "John Doe",
        },
    )

    # Second tool: process the refund
    builder.add_tool_call(
        name="process_refund",
        args={"order_id": order_id, "amount": 89.99},
        result={"refund_id": "RFD-001", "status": "processed"},
    )

    # Set metadata with sample values (no actual LLM API calls)
    builder.set_metadata(
        total_tokens=150,
        cost_usd=0.005,
        latency_ms=1200,
        model="gpt-4",
    )

    # Return the final output
    refund_message = (
        f"Your refund of $89.99 has been processed successfully. "
        f"Refund ID: RFD-001. The funds will appear in your account within 3-5 business days."
    )

    return {
        "message": refund_message,
        "structured": {
            "refund_id": "RFD-001",
            "amount": 89.99,
            "status": "processed",
        },
    }

Write Your First Test

Attest includes a pytest plugin. Re-export its fixtures from conftest.py:

quickstart/conftest.py

"""Pytest configuration for quickstart example."""

from __future__ import annotations

# Re-export the attest fixtures from attest.plugin
from attest.plugin import attest, attest_engine

__all__ = ["attest_engine", "attest"]

Then chain assertions with expect() and evaluate them with the attest fixture:

quickstart/test_agent.py

"""Tests for the customer support agent."""

from __future__ import annotations

from agent import customer_support
from attest.expect import expect


def test_customer_support_refund(attest) -> None:
    """Test the customer support agent processes refunds correctly.

    Demonstrates Attest assertions across all layers:
    - Layer 1: Schema validation
    - Layer 2: Cost and performance constraints
    - Layer 3: Trace structure (tool ordering)
    - Layer 4: Content validation
    """
    # Run the agent
    result = customer_support(order_id="ORD-12345")

    # Build and evaluate assertions using the expect DSL
    chain = (
        expect(result)
        # Layer 1: Schema validation
        .output_matches_schema(
            {
                "type": "object",
                "properties": {
                    "refund_id": {"type": "string"},
                    "amount": {"type": "number"},
                    "status": {"type": "string"},
                },
                "required": ["refund_id", "status"],
            }
        )
        # Layer 2: Cost and performance constraints
        .cost_under(0.01)
        .latency_under(5000)
        .tokens_under(500)
        # Layer 3: Trace structure (tool ordering and requirements)
        .tools_called_in_order(["lookup_order", "process_refund"])
        .required_tools(["lookup_order", "process_refund"])
        .forbidden_tools(["delete_order", "cancel_refund"])
        # Layer 4: Content validation
        .output_contains("refund")
        .output_contains("processed")
        .output_not_contains("error")
    )

    # Evaluate all assertions
    attest.evaluate(chain)

Run Tests

uv run pytest -v

attest.evaluate(chain) fails the test on a hard failure and skips it when an assertion could not be evaluated, naming the reason.

Multiple Trials

Collect each trial with attest.inspect(chain), which never fails or skips the test, and enforce only the aggregate with attest.enforce(...). See Verdicts.

Using Adapters

Use built-in adapters for framework-specific trace capture (requires the openai extra):

This snippet is hand-written and not executed in CI (it needs a provider key); the synced examples above are the executed ones.

from openai import OpenAI

from attest import AgentResult, OpenAIAdapter, expect

client = OpenAI()
adapter = OpenAIAdapter(agent_id="assistant")


def test_explains_recursion(attest):
    messages = [{"role": "user", "content": "Explain recursion"}]
    response = client.chat.completions.create(model="gpt-4.1", messages=messages)

    trace = adapter.trace_from_response(response, input_messages=messages)
    result = AgentResult(trace=trace)

    attest.evaluate(expect(result).output_contains("recursion").tokens_under(1000))

Simulation Mode

Simulation makes embedding and judge assertions report not_evaluated(simulation) while deterministic layers still evaluate. It does not stop provider calls in your own application; use mocks or strict replay for those:

ATTEST_SIMULATION=1 uv run pytest -v

Environment Variables

Variable	Purpose
OPENAI_API_KEY	OpenAI API key for judge/embedding assertions
ANTHROPIC_API_KEY	Anthropic API key for judge assertions
ATTEST_ENGINE_PATH	Override engine binary location
ATTEST_SIMULATION	Skip embedding and judge provider calls; application calls still need mocks
ATTEST_JUDGE_PROVIDER	LLM judge provider: openai, anthropic, gemini, acp
ATTEST_JUDGE_MODEL	Model for judge assertions (e.g., gpt-4.1)

Next Steps

• Expect DSL Reference — All assertion methods
• Adapters Reference — Provider integrations
• Simulation Guide — Simulation runtime details
• Multi-Agent Guide — Testing multi-agent systems