Writing a Plugin

A plugin checks a trace using your own rules. Use one when the built-in assertions do not cover a domain-specific requirement. The SDK runs registered assertion plugins and submits their results before asking the engine to evaluate the batch; you do not need to manage JSON-RPC requests or assertion IDs.

Python: Create a Plugin

In an existing uv project, install Attest and pytest:

uv add "attest-ai>=1.1.0" pytest

Create word_limit.py:

from typing import Any

from attest import Trace
from attest.plugins import PluginResult


class WordLimitPlugin:
    name = "word-limit"
    plugin_type = "assertion"

    def execute(self, trace: Trace, spec: dict[str, Any]) -> PluginResult:
        limit = int(spec.get("config", {}).get("max_words", 50))
        message = trace.output.get("message", "")
        if not isinstance(message, str):
            return PluginResult("hard_fail", 0.0, "output.message must be a string")
        count = len(message.split())
        passed = count <= limit
        return PluginResult(
            status="pass" if passed else "hard_fail",
            score=1.0 if passed else 0.0,
            explanation=f"{count} words; limit is {limit}",
        )


plugin = WordLimitPlugin()

Python plugins use a synchronous execute(trace, spec) method. The SDK runs it in a worker thread with a 30-second timeout. spec is the whole assertion specification: configuration passed to .plugin() is nested under spec["config"].

PluginResult requires status, score, and explanation; it also accepts optional metadata. Metadata is available on the returned plugin result, but the execution helper submits only status, score, and explanation to the engine. Scores do not automatically select a status: choose the status explicitly.

Register and Evaluate in pytest

Create test_word_limit.py:

from attest import TraceBuilder, expect

from word_limit import plugin


def test_short_answer(attest):
    attest.plugin_registry.register(plugin.name, plugin)
    trace = TraceBuilder().set_output(message="Refund processed.").build()
    result = attest.evaluate(
        expect(trace).output_contains("Refund").plugin(
            "word-limit", {"max_words": 10}
        )
    )
    assert result.passed

Run it without a provider or paid model call:

uv run pytest test_word_limit.py

expect() accepts a Trace or an AgentResult(trace=trace). The fixture assigns assertion IDs, looks up the plugin under type "assertion", executes it, submits the result, and evaluates the chain. Register the instance, not the class. Using the same registry key and plugin.name works in both SDKs.

To test the plugin logic without the engine, call it directly:

def test_word_limit_logic():
    trace = TraceBuilder().set_output(message="one two three").build()
    result = plugin.execute(trace, {"config": {"max_words": 2}})
    assert result.status == "hard_fail"

Add this function to the same test module.

Discover an Installed Python Plugin

For a reusable distribution, export the instance from an importable module and declare an entry point in that distribution’s pyproject.toml:

[project.entry-points."attest.plugins"]
word-limit = "word_limit:plugin"

Ensure your package build includes word_limit.py. Install the distribution into the test environment, for example with uv add ../word-limit-plugin. The pytest fixture automatically loads the attest.plugins group at construction, so the test no longer needs the explicit registration line. The entry-point name becomes the registry key; it should match the ID used by .plugin().

For a standalone registry, load_entrypoint_plugins(registry) and register_plugin(registry, name, plugin) are available from attest.plugins. A standalone registry is not automatically the pytest fixture’s registry.

TypeScript: Create and Register a Plugin

In an existing npm project:

npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"

Create word-limit.ts:

import type { AttestPlugin } from "@attest-ai/core";

export const wordLimit: AttestPlugin = {
  name: "word-limit",
  pluginType: "assertion",
  execute(trace, spec) {
    const config = spec["config"] as { maxWords?: number } | undefined;
    const limit = config?.maxWords ?? 50;
    const message = trace.output["message"];
    if (typeof message !== "string") {
      return { status: "hard_fail", score: 0, explanation: "output.message must be a string" };
    }
    const count = message.trim() === "" ? 0 : message.trim().split(/\s+/).length;
    const passed = count <= limit;
    return {
      status: passed ? "pass" : "hard_fail",
      score: passed ? 1 : 0,
      explanation: `${count} words; limit is ${limit}`,
    };
  },
};

TypeScript uses pluginType, not Python’s plugin_type. execute() may return a PluginResult object or a promise of one. Keep registry keys equal to the plugin’s name: the TypeScript SDK rejects a mismatch.

Create word-limit.test.ts:

import { TraceBuilder, attestExpect } from "@attest-ai/core";
import { attestPlugins, evaluate } from "@attest-ai/vitest";
import { test } from "vitest";
import { wordLimit } from "./word-limit";

attestPlugins.register(wordLimit.name, wordLimit);

test("short answer", async (context) => {
  const trace = new TraceBuilder().setOutput({ message: "Refund processed." }).build();
  await evaluate(attestExpect(trace).plugin("word-limit", { maxWords: 10 }), { context });
});

Download and check the engine before the first test run, then run Vitest:

npx attest doctor
npx vitest run word-limit.test.ts

On a cold home directory, the first engine download can exceed Vitest’s default five-second test timeout. Running doctor first avoids putting that download inside the first test.

Discover an Installed TypeScript Plugin

A plugin distribution declares JavaScript module paths in its own package.json:

{
  "name": "word-limit-plugin",
  "version": "1.0.0",
  "type": "module",
  "attest": {
    "plugins": ["./dist/word-limit.js"]
  }
}

Build and include that JavaScript file in the published package; it can export wordLimit as above or export a plugin instance as its default. Install the package as a direct dependency, dev dependency, or optional dependency of the test project. The loader checks installed direct dependencies, not arbitrary source files or transitive packages. Declared module paths must stay inside their package, including after resolving symlinks.

Unlike the Python pytest fixture, the Vitest registry does not discover packages automatically in 1.1.0. Replace explicit registration with a loader call before evaluation (for example, at the top level of the test module):

import { loadPackagePlugins } from "@attest-ai/core";
import { attestPlugins } from "@attest-ai/vitest";

await loadPackagePlugins(attestPlugins);

loadPackagePlugins(registry, projectRoot) defaults projectRoot to the current directory. Explicit registration remains useful for local plugins that are not packaged.

Results and Failures

• Return pass when the requirement is met, hard_fail when it is not, or soft_fail for a recorded non-blocking failure.
• A missing registration leaves the assertion not_evaluated with reason missing_evidence; it is not a passing check. Use strict evaluation when missing evidence must fail the test. In CI policy, set not_evaluated: block to block unevaluated assertions; the default policy only warns.
• Loading failures are logged and discovery continues. Check registration if an installed plugin is not evaluated.
• Execution exceptions and timeouts propagate; the SDK does not turn them into passing results. Handle an expected domain error explicitly if it should produce a structured failure, rather than hiding programming errors with a broad catch.
• Keep plugins deterministic where possible. A word count is a length check, not a toxicity or readability assessment.

See the Python Expect DSL and JSON-RPC Protocol references for the assertion and underlying protocol contracts.