Attest Framework

Why Attest?

57% of organizations run AI agents in production without systematic testing. Most testing tools force expensive LLM-as-judge calls for every assertion. Attest takes a different approach: an 8-layer graduated assertion pipeline that starts with free deterministic checks and only escalates to costly LLM evaluation when cheaper layers can’t answer the question.

The 8-Layer Assertion Pipeline

1

Schema Validation

JSON Schema conformance, required fields, type checking

Free

2

Constraint Checking

Value ranges, string patterns, numeric bounds

Free

3

Trace Validation

Step ordering, tool call verification, timing checks

Free

4

Content Matching

Regex patterns, keyword presence, substring matching

Free

5

Embedding Similarity

Semantic similarity via vector embeddings

≪$0.001

6

LLM Judge

Natural language evaluation by a judge model

~$0.001

7

Trace Tree

Multi-agent delegation chains, cross-agent assertions

Free

8

Plugin

Custom evaluation logic via the plugin protocol

Varies

What It Looks Like

Python

from attest import agent, expect

@agent("weather-agent")
async def weather_agent(query: str) -> str:
    # Your agent implementation
    ...

async def test_weather_agent():
    result = await weather_agent("What's the weather in Tokyo?")

    expect(result).to_have_steps(["llm_call", "tool_call"])  # Layer 3: Free
    expect(result).output_to_contain("Tokyo")                 # Layer 4: Free
    expect(result).output_to_be_similar_to(                   # Layer 5: ≪$0.001
        "Current weather conditions in Tokyo"
    )
    expect(result).output_to_satisfy(                         # Layer 6: ~$0.001
        "Provides accurate, helpful weather information"
    )

TypeScript

import { Agent, attestExpect } from '@attest-ai/core';

const weatherAgent = new Agent('weather-agent', async (query: string) => {
  // Your agent implementation
});

test('weather agent returns weather data', async () => {
  const result = await weatherAgent.run('What is the weather in Tokyo?');

  attestExpect(result).toHaveSteps(['llm_call', 'tool_call']);  // Layer 3: Free
  attestExpect(result).outputToContain('Tokyo');                 // Layer 4: Free
  attestExpect(result).outputToBeSimilarTo(                      // Layer 5: ≪$0.001
    'Current weather conditions in Tokyo'
  );
  attestExpect(result).outputToSatisfy(                          // Layer 6: ~$0.001
    'Provides accurate, helpful weather information'
  );
});

Features

8 Assertion Layers

Graduate from free deterministic checks to expensive LLM evaluation. Layers 1-4 are free and instant. Only pay for what you need.

11 Framework Adapters

OpenAI, Anthropic, Gemini, Ollama, LangChain, LlamaIndex, CrewAI, Google ADK, OpenTelemetry - plug in your stack.

Multi-Agent Testing

Trace delegation chains across agents. Assert on parent-child relationships, temporal ordering, and cross-agent data flow.

Simulation Mode

Test without API calls. Mock tools, inject faults, simulate adversarial users - all deterministic and free.

Continuous Evaluation

Run assertions on production traces. Sample, evaluate, and alert on drift - Slack, webhooks, custom dispatchers.

Cost Tracking

Know exactly what each test costs. Set tier budgets, soft failure thresholds, and cost-per-assertion metrics.

How It Compares

Feature	Attest	DeepEval	PromptFoo	Ragas	LangWatch
Deterministic assertions	✓	✗	✓	✗	✗
Graduated cost layers	✓	✗	✗	✗	✗
LLM-as-judge	✓	✓	✓	✓	✓
Multi-agent traces	✓	✗	✗	✗	✗
Simulation mode	✓	✗	✗	✗	✗
Framework adapters (5+)	✓	✗	✗	✓	✓
Code-first (not config)	✓	✓	✗	✓	✗
Python + TypeScript SDKs	✓	✗	✓	✗	✗

Get Started

# Python
pip install attest-ai

# TypeScript
npm install @attest-ai/core @attest-ai/vitest

Check the Python quickstart or TypeScript quickstart for a complete walkthrough.