Migrating from Promptfoo

Use Attest 1.1.0 when you want checks inside pytest or Vitest, including checks over tool calls and external effects. Migrate the intent of each test case, not just its configuration syntax. Existing prompt evaluations can remain alongside the new tests.

A minimal code-first check

uv add "attest-ai>=1.1.0" pytest

Save this as test_qa.py and run uv run pytest. It uses sample data, not a model API:

from attest import TraceBuilder, expect


def test_arithmetic_reply(attest):
    trace = TraceBuilder("qa").set_output(message="4").build()
    attest.evaluate(expect(trace).output_contains("4").output_matches_regex(r"^\d+$"))

For a real provider response, use an adapter instead of the sample trace. The adapter normalizes a response; your application still makes the provider request.

Convert one case at a time

Existing test intent	Attest approach
Input variables and expected result	A pytest parameterized case or a Vitest test
Contains / regex requirement	output_contains / output_matches_regex in Python; camelCase in TypeScript
JSON output shape	output_matches_schema over structured output
Cost or latency budget	cost_under / latency_under, using recorded usage and timing
Subjective rubric	passes_judge with a configured and calibrated provider/model
Tool order, per-key limits or final external state	Behavior contracts and world fixtures

Thresholds are application requirements, not provider price estimates. Do not hard-code invented usage values into a real trace to make a budget check pass. Judge scores from different tools are not necessarily comparable; recalibrate before enforcing them.

For offline regression checks, capture a run, review its contracts and use strict replay. Matching requests use recorded model responses while the agent’s real code runs. Mismatches do not silently call a live model in strict mode.

Finish with a deliberately failing case, then configure the merge gate. Unevaluated assertions only warn under the default policy; choose not_evaluated: block when missing measurements should block. See the Python and TypeScript setup pages for runner integration.