Quickstart

Attest 1.1.0 checks what your agent says and does inside pytest or Vitest. These examples record sample data without calling a model. The code is imported from the quickstart and quickstart-ts examples, which CI runs on every change.

Installation

Python

uv add "attest-ai>=1.1.0" pytest

TypeScript

npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"

Write Your First Test

Step 1: Create an Agent

The example records a refund request: one sample LLM call, then lookup_order and process_refund tool calls. The agent decorator supplies a TraceBuilder; the example explicitly records the steps so the test can assert on the trace without calling a model. Its cost metadata and budget thresholds are illustrative, not provider price claims.

Python

quickstart/agent.py

"""Simple customer support agent for Attest quickstart example."""

from __future__ import annotations

from typing import Any

from attest.agent import agent
from attest.trace import TraceBuilder


@agent("customer-support")
def customer_support(builder: TraceBuilder, order_id: str) -> dict[str, Any]:
    """Customer support agent that processes refund requests.

    Args:
        builder: TraceBuilder for constructing the trace
        order_id: The order ID to look up and potentially refund

    Returns:
        A dictionary with the refund result
    """
    # Simulate LLM call to understand the customer's request
    builder.add_llm_call(
        name="gpt-4",
        args={
            "messages": [
                {
                    "role": "user",
                    "content": f"Customer wants to return order {order_id}. Process the refund.",
                }
            ],
        },
        result={"response": "I will process the refund for this order."},
    )

    # First tool: look up the order details
    builder.add_tool_call(
        name="lookup_order",
        args={"order_id": order_id},
        result={
            "order_id": order_id,
            "status": "delivered",
            "amount": 89.99,
            "customer": "John Doe",
        },
    )

    # Second tool: process the refund
    builder.add_tool_call(
        name="process_refund",
        args={"order_id": order_id, "amount": 89.99},
        result={"refund_id": "RFD-001", "status": "processed"},
    )

    # Set metadata with sample values (no actual LLM API calls)
    builder.set_metadata(
        total_tokens=150,
        cost_usd=0.005,
        latency_ms=1200,
        model="gpt-4",
    )

    # Return the final output
    refund_message = (
        f"Your refund of $89.99 has been processed successfully. "
        f"Refund ID: RFD-001. The funds will appear in your account within 3-5 business days."
    )

    return {
        "message": refund_message,
        "structured": {
            "refund_id": "RFD-001",
            "amount": 89.99,
            "status": "processed",
        },
    }

TypeScript

quickstart-ts/agent.ts

/**
 * Simple customer support agent for Attest quickstart example.
 *
 * TypeScript port of the Python quickstart — same agent, same trace structure.
 */
import { agent, type TraceBuilder } from "@attest-ai/core";

export const customerSupport = agent(
  "customer-support",
  (builder: TraceBuilder, args: Record<string, unknown>) => {
    const orderId = args.order_id as string;

    // Simulate LLM call to understand the customer's request
    builder.addLlmCall("gpt-4", {
      args: {
        messages: [
          {
            role: "user",
            content: `Customer wants to return order ${orderId}. Process the refund.`,
          },
        ],
      },
      result: { response: "I will process the refund for this order." },
    });

    // First tool: look up the order details
    builder.addToolCall("lookup_order", {
      args: { order_id: orderId },
      result: {
        order_id: orderId,
        status: "delivered",
        amount: 89.99,
        customer: "John Doe",
      },
    });

    // Second tool: process the refund
    builder.addToolCall("process_refund", {
      args: { order_id: orderId, amount: 89.99 },
      result: { refund_id: "RFD-001", status: "processed" },
    });

    // Set metadata
    builder.setMetadata({
      total_tokens: 150,
      cost_usd: 0.005,
      latency_ms: 1200,
      model: "gpt-4",
    });

    return {
      message: `Your refund of $89.99 has been processed successfully. Refund ID: RFD-001. The funds will appear in your account within 3-5 business days.`,
      structured: {
        refund_id: "RFD-001",
        amount: 89.99,
        status: "processed",
      },
    };
  },
);

Step 2: Add Assertions

Chain assertions on the agent result, then evaluate the chain. In Python, the pytest plugin provides the attest fixture; re-export it from conftest.py:

Python

quickstart/conftest.py

"""Pytest configuration for quickstart example."""

from __future__ import annotations

# Re-export the attest fixtures from attest.plugin
from attest.plugin import attest, attest_engine

__all__ = ["attest_engine", "attest"]

quickstart/test_agent.py

"""Tests for the customer support agent."""

from __future__ import annotations

from agent import customer_support
from attest.expect import expect


def test_customer_support_refund(attest) -> None:
    """Test the customer support agent processes refunds correctly.

    Demonstrates Attest assertions across all layers:
    - Layer 1: Schema validation
    - Layer 2: Cost and performance constraints
    - Layer 3: Trace structure (tool ordering)
    - Layer 4: Content validation
    """
    # Run the agent
    result = customer_support(order_id="ORD-12345")

    # Build and evaluate assertions using the expect DSL
    chain = (
        expect(result)
        # Layer 1: Schema validation
        .output_matches_schema(
            {
                "type": "object",
                "properties": {
                    "refund_id": {"type": "string"},
                    "amount": {"type": "number"},
                    "status": {"type": "string"},
                },
                "required": ["refund_id", "status"],
            }
        )
        # Layer 2: Cost and performance constraints
        .cost_under(0.01)
        .latency_under(5000)
        .tokens_under(500)
        # Layer 3: Trace structure (tool ordering and requirements)
        .tools_called_in_order(["lookup_order", "process_refund"])
        .required_tools(["lookup_order", "process_refund"])
        .forbidden_tools(["delete_order", "cancel_refund"])
        # Layer 4: Content validation
        .output_contains("refund")
        .output_contains("processed")
        .output_not_contains("error")
    )

    # Evaluate all assertions
    attest.evaluate(chain)

TypeScript

quickstart-ts/agent.test.ts

/**
 * Tests for the customer support agent.
 *
 * Demonstrates Attest assertions across layers 1-4:
 * - Layer 1: Schema validation
 * - Layer 2: Cost and performance constraints
 * - Layer 3: Trace structure (tool ordering)
 * - Layer 4: Content validation
 */
import { attestExpect } from "@attest-ai/core";
import { evaluate } from "@attest-ai/vitest";
import { describe, it, expect } from "vitest";
import { customerSupport } from "./agent";

describe("customer support agent", () => {
  it("processes refunds correctly", async () => {
    const result = customerSupport({ order_id: "ORD-12345" });

    const chain = attestExpect(result)
      // Layer 1: Schema validation
      .outputMatchesSchema({
        type: "object",
        properties: {
          refund_id: { type: "string" },
          amount: { type: "number" },
          status: { type: "string" },
        },
        required: ["refund_id", "status"],
      })
      // Layer 2: Cost and performance constraints
      .costUnder(0.01)
      .latencyUnder(5000)
      .tokensUnder(500)
      // Layer 3: Trace structure
      .toolsCalledInOrder(["lookup_order", "process_refund"])
      .requiredTools(["lookup_order", "process_refund"])
      .forbiddenTools(["delete_order", "cancel_refund"])
      // Layer 4: Content validation
      .outputContains("refund")
      .outputContains("processed")
      .outputNotContains("error");

    const evaluated = await evaluate(chain);
    expect(evaluated.passed).toBe(true);
  });
});

Step 3: Run Tests

Python

uv run pytest -v

TypeScript

Warm up the engine before the first Vitest run: on a cold machine its initial download can exceed the default five-second test timeout.

npx attest doctor
npx vitest run

A failing assertion fails the test and names the assertion. Swap the tool order in tools_called_in_order / toolsCalledInOrder to see it.

Understanding Assertions

Attest uses a fluent API where you chain assertions. Each assertion validates a different aspect:

Layer	Python	TypeScript	What it checks
Schema	.output_matches_schema({...})	.outputMatchesSchema({...})	Output structure
Constraints	.cost_under(0.10), .latency_under(2000)	.costUnder(0.10), .latencyUnder(2000)	Performance metrics
Trace	.tools_called_in_order([...])	.toolsCalledInOrder([...])	Execution path
Content	.output_contains("hello")	.outputContains("hello")	Text content
Embedding	.output_similar_to("greeting")	.outputSimilarTo("greeting")	Semantic meaning
LLM Judge	.passes_judge("Is this polite?")	.passesJudge("Is this polite?")	Domain-specific eval
Trace Tree	.agent_called("billing")	.agentCalled("billing")	Multi-agent structure
Plugin	.plugin("my-plugin", {...})	.plugin("my-plugin", {...})	Custom checks

Layers 1–4, including trace-tree and effect checks in layer 3, are deterministic and free. Embedding and judge assertions use a configured provider whose cost varies; under ATTEST_SIMULATION=1 they return not_evaluated(simulation) instead. Simulation does not intercept your application’s own network calls.

Next Steps

• Python Setup and TypeScript Setup — Project setup per language
• Python SDK Reference — Explore all assertion methods
• Adapters — Learn about provider integration
• Migration Guides — Upgrade from other frameworks
• Write an Adapter — Build a custom integration

Common Patterns

Multiple Trials

attest.evaluate(chain) fails or skips the test on its own. To judge several runs together, collect each with attest.inspect(chain) and enforce only the aggregate; see Verdicts.

Adapter Integration

Use built-in adapters to build a trace from a provider response:

This snippet is hand-written and not executed in CI (it needs a provider key); the synced examples above are the executed ones.

from openai import OpenAI

from attest import AgentResult, OpenAIAdapter, expect

client = OpenAI()
adapter = OpenAIAdapter(agent_id="assistant")


def test_explains_recursion(attest):
    messages = [{"role": "user", "content": "Explain recursion"}]
    response = client.chat.completions.create(model="gpt-4.1", messages=messages)
    result = AgentResult(trace=adapter.trace_from_response(response, input_messages=messages))
    attest.evaluate(expect(result).output_contains("recursion").tokens_under(1000))

Troubleshooting

“API key not found”

Hosted embedding and judge providers need their own configuration and credentials. Local ONNX embeddings do not use a provider key; local ACP judges use the agent’s login and are refused in CI. See configuration and judges rather than adding keys to test files.

“Assertion failed but I expected it to pass”

Inspect the result the agent produced:

print(result.trace.output)    # What the agent returned
print(result.trace.steps)     # LLM and tool calls in order
print(result.trace.metadata)  # Tokens, cost and latency

“Tests call a paid model”

ATTEST_SIMULATION=1 prevents Attest’s embedding and judge provider calls; those assertions report not_evaluated(simulation). Your application’s own model calls still run unless you replace them with mocks or strict replay.

Learn More

Check the API reference for complete method documentation and the migration guides if you’re upgrading from another framework.