Simulation Runtime

Simulation helps you test retries, errors and conversations under controlled conditions. The runtime is beta: it provides mocked tools, seeded fault injection, repeated runs and user personas. It does not automatically replace network calls made by your own application.

Run deterministic checks without a judge

ATTEST_SIMULATION=1 uv run pytest -v

This makes embedding and judge assertions return not_evaluated with reason simulation; it does not pretend they passed. Schema, constraints, trace and content checks still evaluate. Tests that call live providers themselves need mocks or replay as well.

Choose the right tool

• Use scenario for mock tools, faults and multi-turn orchestration.
• Use repeat with evaluated trial results to measure repeated outcomes. A trace alone is not evidence that assertions passed.
• Use strict replay to run real agent code with recorded model responses.
• Use world fixtures to check effects in an isolated external state.

The SDK simulation guide contains the decorators, persona examples and result semantics. Statistical verdicts explains how to enforce an aggregate without treating unevaluated trials as passes. Model-generated user turns still need a configured provider and may incur cost; mocks and seeded faults alone do not call one.