TypeScript Expect DSL

Use attestExpect() to describe the output, tool calls and effects you expect from an agent. Building a chain only collects checks: pass it to evaluate() to run them and enforce the test result. The API mirrors the Python DSL with camelCase methods and option objects.

Installation and Minimal Test

npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"
npx attest doctor
npx vitest run

On a cold HOME, run npx attest doctor before the first test: downloading the engine can exceed Vitest’s default five-second first-test timeout. Configure the Vitest integration for reporting and lifecycle checks.

import { TraceBuilder, attestExpect } from '@attest-ai/core';
import { evaluate } from '@attest-ai/vitest';
import { it } from 'vitest';

it('greets the user', async (context) => {
  const trace = new TraceBuilder('greeter').setOutput({ message: 'Hello' }).build();
  await evaluate(attestExpect(trace).outputContains('hello'), { context });
});

Chains and Evaluation

attestExpect(result: AgentResult | Trace): ExpectChain accepts a result or raw trace. It wraps a raw trace in an AgentResult. Assertion methods return the chain.

Property	Type	Meaning
assertions	Assertion[]	Copy of collected assertions
trace	Trace	Underlying trace
evaluated	boolean	The current assertion list was submitted for evaluation

Import evaluate, inspect, and useAttest from @attest-ai/vitest.

Function	Behavior
await evaluate(chain, options?)	Evaluate, then enforce; returns AgentResult if the test continues
await inspect(chain, options?)	Evaluate without enforcing the assertion outcome
useAttest().enforce(verdict, options?)	Enforce an inspected result or aggregate trial verdict

inspect options are budget?: number (cumulative session-cost limit in USD), testBudget?: { maxJudgeCalls?: number; maxCostUsd?: number } (this test’s model-call limits), and context?: TestContext. evaluate additionally accepts strict?: boolean, defaulting to ATTEST_STRICT (1, true, or yes). Enforcement accepts strict and context. Exceeding the session budget can still throw from inspect().

Any hard_fail fails. Otherwise not_evaluated skips (fails under strict); otherwise the test passes, including soft failures. Pass { context } under it.concurrent and for skips on Vitest before 4.1. During a trial of trials() with multiple runs, enforcement is deferred to the aggregate verdict.

In 1.1.0 the configured Vitest integration fails an otherwise passing test that leaves an assertion chain unevaluated. This includes assertions appended after evaluation and chains in concurrent tests. Submit the final chain with evaluate() or inspect(); inspecting is not the same as enforcing. Evaluation assigns stable assertion IDs (scheme 2) from test scope, type, specification and occurrence rather than the position among unrelated checks.

Layer 1: Schema

Schema methods do not accept a soft option.

Method	Parameters	Target
outputMatchesSchema	schema: Record<string, unknown>	output.structured, not the whole output
outputFieldMatchesSchema	field: string, schema: Record<string, unknown>	output.<field>
toolArgsMatchSchema	toolName: string, schema: Record<string, unknown>	Named steps’ arguments
toolResultMatchesSchema	toolName: string, schema: Record<string, unknown>	Named steps’ results

it('returns structured output', async (context) => {
  const trace = new TraceBuilder().setOutput({ structured: { answer: 42 } }).build();
  await evaluate(attestExpect(trace).outputMatchesSchema({
    type: 'object', required: ['answer'],
    properties: { answer: { type: 'integer' } },
  }), { context });
});

Layer 2: Constraints

Each method accepts a final optional { soft?: boolean } object, defaulting to false, unless a fuller option object is shown.

Method	Parameters	Check
costUnder	maxCost: number	metadata.cost_usd <= maxCost
latencyUnder	maxMs: number	metadata.latency_ms <= maxMs
tokensUnder	maxTokens: number	metadata.total_tokens <= maxTokens
tokensBetween	minTokens: number, maxTokens: number	Inclusive range
stepCount	operator: string, value: number	Number of steps
toolCallCount	operator: string, value: number	Number of tool_call steps
constraint	field: string, operator: string, opts?: { value?: number; min?: number; max?: number; soft?: boolean }	Generic numeric constraint

Use eq, lt, lte, gt, or gte with a value; use between with constraint’s min and max. Missing metrics are not measured zeroes: populate the metadata a constraint needs.

const chain = attestExpect(trace)
  .costUnder(0.10)
  .tokensBetween(100, 500)
  .constraint('metadata.latency_ms', 'lte', { value: 5000, soft: true });

Layer 3: Trace Checks

The original trace checks match all step names, not only tools. They accept final opts?: { soft?: boolean }.

Method	Parameters	Check
toolsCalledInOrder	tools: string[]	Ordered subsequence; other steps may occur between
toolsCalledExactly	tools: string[]	Contiguous subsequence; other steps may occur before/after
requiredTools	tools: string[]	All names occur
forbiddenTools	tools: string[]	No listed name occurs
noToolLoops	tool: string, maxRepetitions = 1	Bound total occurrences of a name, not just consecutive repeats
noDuplicateTools	No positional arguments	No step name repeats

Behavior Contracts

These methods examine only tool_call steps; retries count as calls.

Method	Parameters after the method name
toolCardinality	(tool: string, opts: { per?: string; atMost?: number; atLeast?: number; soft?: boolean })
trajectory	(tools: string[], opts?: { mode?: "strict" | "unordered" | "subset" | "superset"; soft?: boolean })
trajectoryPartialOrder	(order: [string, string][], opts?: { tools?: string[]; soft?: boolean })
toolCalledBefore	(before: string, after: string, opts?: { soft?: boolean })
noToolCycles	(period: number, opts?: { minRepeats?: number; soft?: boolean })
maxToolFailureStreak	(maxStreak: number, opts?: { soft?: boolean })
toolArgSatisfies	(tool: string, arg: string, op: "eq" | "ne" | "lt" | "lte" | "gt" | "gte" | "in" | "not_in", value: unknown, opts?: { soft?: boolean })

toolCardinality requires atMost, atLeast, or both, optionally grouped by the per argument path. trajectory defaults to strict (same sequence); unordered means the same calls in any order; subset permits no calls outside the list; superset requires all listed calls. Partial order requires every tool named by its pairs or optional tools, plus the before/after constraints.

toolCalledBefore requires an earlier before for each after, but passes if there is no after. Argument predicates use a dot path into arguments and pass when the tool is absent; add a cardinality minimum when existence matters. noToolCycles checks repeated blocks with a period of 1–4 and defaults to two repeats. Failure streak checks require trace-v2 error evidence. Missing required argument/error evidence is not_evaluated: missing_evidence.

it('issues at most one refund per order', async (context) => {
  const trace = new TraceBuilder('refund')
    .addToolCall('check_eligibility', { args: { order_id: 'O-1042' } })
    .addToolCall('issue_refund', { args: { order_id: 'O-1042', amount_cents: 500 } })
    .setOutput({ message: 'Refund issued' }).build();
  await evaluate(attestExpect(trace)
    .toolCardinality('issue_refund', { per: 'order_id', atMost: 1 })
    .toolCalledBefore('check_eligibility', 'issue_refund')
    .toolArgSatisfies('issue_refund', 'amount_cents', 'lte', 500), { context });
});

Layer 3: World and Effect Contracts

A tool call is an attempt; a world snapshot shows the resulting state. chain.world(snapshot: Record<string, unknown> | null) returns WorldChecks. Each effect check returns the parent chain. Obtain the snapshot from await world.state() or await world.stop() with the world helper; null becomes not_evaluated: missing_evidence.

Method	Parameters
count	(resource: string, opts: { equals?: number; atMost?: number; atLeast?: number; where?: Record<string, unknown> })
unique	(resource: string, opts: { by: string | string[]; where?: Record<string, unknown> })
idempotent	(resource: string, opts?: { key?: string; where?: Record<string, unknown> })
sum	(resource: string, field: string, opts: { atMost: number | Related; where?: Record<string, unknown> })
matches	(resource: string, field: string, rel: Related, opts?: { where?: Record<string, unknown> })

count needs equals or at least one bound; do not combine equals with another bound. where is an exact-match filter. unique checks distinct values of one or several fields; idempotent defaults to the field idempotency_key. Effects have no soft option.

Import related from @attest-ai/core. related(resource: string, field: string, opts: { on: string }): Related names a record in another resource joined on a shared key. A numeric sum bound checks the filtered total. A related bound checks each join-key group’s total against its corresponding record; matches compares each selected record’s field to its corresponding field.

import { related } from '@attest-ai/core';

// In your test, after obtaining the world's snapshot:
const chain = attestExpect(trace);
chain.world(snapshot).count('refunds', { equals: 1, where: { order_id: 'O-1042' } });
chain.world(snapshot).unique('refunds', { by: 'idempotency_key' });
chain.world(snapshot).idempotent('refunds');
chain.world(snapshot).sum('refunds', 'amount_cents', {
  atMost: related('orders', 'amount_cents', { on: 'order_id' }),
});
chain.world(snapshot).matches('notifications', 'customer_id',
  related('orders', 'customer_id', { on: 'order_id' }));
await evaluate(chain, { context });

Invalid snapshots, absent resources, non-numeric sum evidence and ambiguous related keys fail rather than pass; a missing corresponding record is a relational violation. Empty selected records can satisfy a bound: add count when records must exist. The SDK rejects chains whose repeated snapshot payloads exceed 9 MiB.

Layer 4: Content

Output helpers target output.message. Matching defaults to case-insensitive. Each method returns the chain.

Method	Parameters
outputContains	(value: string, opts?: { caseSensitive?: boolean; soft?: boolean })
outputNotContains	(value: string, opts?: { caseSensitive?: boolean; soft?: boolean })
outputMatchesRegex	(pattern: string, opts?: { soft?: boolean })
outputHasAllKeywords	(keywords: string[], opts?: { caseSensitive?: boolean; soft?: boolean })
outputHasAnyKeyword	(keywords: string[], opts?: { caseSensitive?: boolean; soft?: boolean })
outputForbids	(terms: string[]); no soft option
contentContains	(target: string, value: string, opts?: { caseSensitive?: boolean; soft?: boolean })

const chain = attestExpect(trace)
  .outputContains('refund')
  .outputForbids(['password', 'ssn'])
  .contentContains('output.message', 'issued');

Layers 5–6: Semantic Checks

outputSimilarTo(reference: string,
  opts?: { threshold?: number | 'dynamic'; model?: string; soft?: boolean }): this
passesJudge(criteria: string,
  opts?: { rubric?: string; threshold?: number | 'dynamic'; model?: string; soft?: boolean }): this

Both target output.message, default threshold to 0.8 and soft to false. The judge’s default rubric is "default"; model overrides the provider default. Layer 5 requires an embedding provider (local ONNX or hosted OpenAI); Layer 6 requires a judge. Hosted calls can cost money. See Configuration for providers, timeouts, request and USD budgets.

const chain = attestExpect(trace)
  .outputSimilarTo('The refund was issued', { threshold: 0.85 })
  .passesJudge('The answer clearly explains the refund', { rubric: 'default' });

An abstaining judge, malformed reply, rejected credential or provider failure is not_evaluated, never an invented passing score. Only transient failures are retried. Simulation evaluates deterministic checks but returns not_evaluated: simulation for Layers 5–6. Strict mode fails rather than skips unevaluated tests; for report policy, change the default not_evaluated: warn to not_evaluated: block when CI must reject those rows.

threshold: 'dynamic' uses the latest 50 passing or soft-failing scores for the stable assertion ID. With at least 10 scores the bar is max(0.8, mean − 2·max(σ, 0.02)); otherwise 0.8. Pre-1.1 ordinal-ID history does not match the new stable IDs, so baselines rebuild. See Drift Detection.

Trace Tree Checks (Layer 3)

These operate on nested agent_call steps. All accept final opts?: { soft?: boolean }, except that agentOutputContains also supports caseSensitive?: boolean in that object.

Method	Parameters before options
agentCalled	agentId: string
delegationDepth	maxDepth: number (root depth 0)
agentOutputContains	agentId: string, value: string
crossAgentDataFlow	fromAgent: string, toAgent: string, field: string
followsTransitions	transitions: [string, string][]
aggregateCostUnder	maxCost: number
aggregateTokensUnder	maxTokens: number
agentOrderedBefore	agentA: string, agentB: string
agentsOverlap	agentA: string, agentB: string
agentWallTimeUnder	agentId: string, maxMs: number
orderedAgents	groups: string[][]

Ordering and overlap need recorded step timestamps. Use TraceBuilder and TraceTree to construct and inspect nested traces.

Plugins

plugin(pluginId: string, config?: Record<string, unknown>, opts?: { soft?: boolean }): this invokes a registered assertion plugin. The built-in plugin helpers aggregateLatencyUnder(maxMs, opts?) and allToolsCalled(toolNames, opts?) check summed trace-tree latency and tools across the whole tree; both accept { soft?: boolean }.

const chain = attestExpect(trace)
  .aggregateLatencyUnder(5000)
  .allToolsCalled(['search', 'summarize'])
  .plugin('my_plugin', { threshold: 0.9 });

See Writing a Plugin for registration and explicit package discovery.

Soft Failures and Results

Only methods that expose soft accept it. Schema, effect methods and outputForbids do not. A soft_fail records a warning without itself failing the test; it does not turn unavailable evidence or providers into a pass. A session soft-failure limit can still fail a batch.

The returned AgentResult exposes outcome(), hardFailures, softFailures, notEvaluated, and diagnostic helpers. passed is stricter than the test verdict: it is true only when every assertion passed. See AgentResult.