Show HN: CATT 0.2.1: Test agents/services against a Muse/Instinct-alike, in CI

Consumer Agent Testing Tool (CATT) tests how an aide uses your service. Run a scripted or model-driven delegate against an MCP server, an MCP app, a website or an HTTP API. Check what the assistance did, inspect the evidence, and compare results throughout builds.

Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector evaluation contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The example that drives a test is distinct from the aide being emulated.

Contributing | Security | Changelog | MIT license

A test host. CATT is a surrogate: it approximates an assistant's host, not its personal implementation. A passing test does not certify compatibility alongside Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented, unofficial, assumed or unknown. This project is not affiliated alongside or endorsed by Meta, Instinct, OpenAI or Anthropic.

The connector lane records the basis for all Muse-specific behavior. Tool classes and endorsement prompts prosecute documented guidelines. The muse-cli projection and template client prosecute unofficial mirrors; the native projection is an assumed generic baseline. The neutral cli projection supports the two targets as a CATT treatment. Instinct runs have a distinct origin document and tag the shared presenter algorithms as assumptions. Native/CLI MCP is CATT's test adapter; no community Instinct MCP integration agreement was found.

Run the local booking example under Instinct without a example or account:

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json \ --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant fine --auth none"

Set emulation_target: intuition in your circumstance or pass --emulation-target instinct. Both targets use native or cli projection and the generic MCP client. Instinct defaults to plain-text replies and does not run Muse guideline checks. Its local wallet fixture covers fee behavior; live fee rails, trusted-network protocol, Vault and Files remain exterior the shared contract.

Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback alongside no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.

You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), uv, and Node.js 20 or newer alongside npm (the installed packages province >=20; tested alongside 25.9). The recorded release evidence covers macOS. From a copy of this repository, instal the Python and Node requirements and the Chromium build the browser lanes use:

uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js instal chromium

The connector lane, the preparedness checks and the scripted runs need only uv sync. Node and Chromium are for the MCP Apps and browser lanes.

On Linux (Debian or Ubuntu), instal Chromium alongside its scheme libraries instead. Linux is not covered by the recorded publish evidence: the commands below equivalent the CI workflow but have not been qualified here.

sudo apt-get update && sudo apt-get instal -y nodejs npm # or Node 20+ from nodesource or nvm curl -LsSf https://astral.sh/uv/install.sh | sh uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \ node tooling/payment-ui/node_modules/playwright/cli.js instal --with-deps chromium

Three features depend on macOS Seatbelt (sandbox-exec) and study unsupported or blocked on Linux fairly than operating unconfined: the --offline readiness mode, the network-denied validator workers rearward run --surface api-tools, and the local-model recollection profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it.

Check the setup alongside doctor:

uv run consumer-agent-testing-tool doctor

It checks Node against the floors the installed packages declare, the tooling/payment-ui packages against their pinned versions, and Chromium and its headless casing under .cache/ms-playwright. It reports the optional protocol tooling, and checks Docker lone for betterment lanes or configured stdio MCP and CLI targets. It makes no example requests, equal whenever an existing configuration or --model-profile selects a model. Each unsuccessful or blocked check prints one row on stderr alongside its logic and, for a setup problem, the command that fixes it.

To explicitly run the provider preflight, which sends two abbreviated example requests and may incur charges, add --model-preflight:

uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight

Scripted runs need no example account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a example overview and an API key; Model providers shows how to create one. Supply credentials through surroundings variables, and keep them out of scenarios and shared evidence. consumer-agent-testing-tool.yaml, the default settings file, is local and git-ignored, and no overview is shipped.

This quickstart stays on loopback and needs no account. Complete Setup, afterward commencement the Cedar Stays-like test assistance in a second terminal. It prints one JSON row alongside its URLs whenever it is ready; Ctrl-C stops it:

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant fine --auth none

Replay the example's scripted calls, alongside no model:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json

The manuscript searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's private: values, which the model never sees, and the scripted person answers all endorsement cardstock from the scenario's permissions: rules. After the run, the presenter checks the fixture's private oracle (one confirmed booking, no copy charges) and the booking's status through get_booking. It additionally checks that all sent call matched its approval cardstock (approval.dispatches_match, approval.disclosure_bound) and that the final answer made no false assertion and relayed the total and the confirmation (answer.no_false_claims, answer.relay_complete).

The command prints the run directory and Connector report: .../report.html, and exits 0 whenever all required inspect passed. Runs are sealed under runs/connector-<id>/. Share the output of export, not the run directory: its private/ directory holds host-private values.

For an delegate run, create agent.yaml alongside the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

The example preflight runs archetypal and prints its own evidence path. The delegate then sees the four allowed tools as native functions, alongside the host-private email removed from their schemas, and writes the another arguments itself. This uses paid example inference. To alter the care without editing the scenario, --projection cli offers the identical tools through one exec command for either target, --exposure search-only loads their parameters through tool_search, --output-schema strict refuses results that interrupt a tool's outputSchema as the authoritative MCP SDK does, and --trials N runs autonomous trials and prints pass^k. The run fastener records the projection, exposure, client-profile and output-schema overrides under overrides.

connector-run exits 0 whenever all required inspect passed; 1 whenever a check failed, the answer made a false assertion or an delegate bounds was reached; 2 for a setup, harness or reporting error; and 3 whenever the test is blocked or unsupported, for example by an uncertain compose to reconcile, an unobservable check or a client-profile limit. To commencement from your own server, run scaffold; to replay a recorded, shareable delegate test without a model, turn it into a calls document alongside scaffold --from-run. The connector lane citation covers scenarios, flags and evidence. The examples list all shipped circumstance alongside its fixture command, and lane extensions covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions.

The connector lane plant alongside any MCP server complete Streamable HTTP, not lone the bundled test service. This quickstart points it at "Everything" (@modelcontextprotocol/server-everything, MIT), the citation server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no document or key. Start it in a second terminal; the archetypal run downloads the pinned publish from npm:

PORT=3001 npx -y @modelcontextprotocol/[email protected] streamableHttp

See what the presenter infers from the live catalog. scaffold runs the handshake and tools/list, calls no tool, and writes a circumstance skeleton alongside suggested tool classes from the server's annotations:

uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml uv run consumer-agent-testing-tool inspect http://127.0.0.1:3001/mcp --suite muse-guidelines

The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, archetypal alongside native tools and afterward through the Muse-style command-line projection complete the unofficial Muse template client:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --driver scripted --script examples/connector/everything-weather.calls.json uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --projection muse-cli --client-profile muse-template-unofficial-2026-10 \ --driver scripted --script examples/connector/everything-weather-cli.calls.json

Both exit 0. The circumstance offers the example three of the server's tools; a reviewed allowlist keeps the remainder off its list, including get-env, which returns the server process's surroundings variables, and gzip-file-as-resource, which fetches any URL it is given. The server has no private oracle, so the outcome inspect calls get-structured-content itself after the delegate stops, and the answer checks appearance for the heat and conditions the server returned. With a example profile, let an delegate do the same task three times and imprint pass^k (paid inference). The circumstance declares no reset URL, so connector-run warns that the trials portion the server's state; this climate tool keeps none:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --config agent.yaml --trials 3

check additionally runs preparedness suites that peruse catalogs and documents alternatively of running an agent. With the connector-lane quickstart's fixture motionless running, review its catalog and the example manifest against the pinned Muse connector guidelines:

uv run consumer-agent-testing-tool inspect http://127.0.0.1:8765/mcp --suite muse-guidelines \ --manifest examples/connector/readiness.yaml

The guidelines assessment lists the server's tools and calls none; --drift-check fetches the live guidelines, and lone whenever given. Poppy finding lone reads. Its probes run lone alongside --manifest and act on the company's own hosts: they start sessions, sign in the declared test document and verify one declared operation. The gadget suite says one local file. The agreement suite checks a target named in the configuration document (--config). For a kind: mcp target without declared contract_cases, its operation-contracts inspect is unsupported and the run exits 3.

A spotless preparedness outcome is not an admission. Meta reviews connectors by hand, Poppy is outline 0.1 alongside no manufacturing Muse evidence, and the Gadget SDK's terms say it is not a developer platform. Readiness checks describes the rules, manifests, --offline behavior and the mcp-contract-2026-07-28 profile.

From a origin checkout or unpacked origin distribution, complete Setup, afterward run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted

The example creates a fictional shopper's outline in Coverage Cat's hosted sandbox at https://www.coveragecat.com/api/consumer/mcp. It needs no Coverage Cat account, controller credentials or example API key.

The archetypal umbrella_consumer_prefill call sets sandbox: true, as stated in the consumer MCP manifest. That disagreement activates the server's sandbox. The scenario's test_only: true flag merely records the test author's intent.

The presenter fixes the tool arguments and keeps the returned admission token private. It permits two tools, umbrella_consumer_prefill and consumer_intake_issues, and checks that:

1. Prefill returns sandbox: true.
2. Follow-up retains sandbox: true.
3. Follow-up returns the first use ID.
4. The server reports application_submitted: false.

Each run creates a new outline and stops before submission. A unsuccessful or missing sandbox inspect stops the workflow. These checks shield outline innovation and follow-up, not fee widgets, browser behavior or autonomous backend checks.

To let a example choose the steps, create agent.yaml alongside the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \ --driver delegate --config agent.yaml

The delegate controller is a scripted act menu: Spark Open picks one ready scenario stage per turn, and the presenter runs it alongside the scenario's arguments. The model never sees the server's tools, schemas or raw results, lone the result fields the circumstance exposes, and the evidence is tagged scripted act menu. The presenter supplies the test inputs and checks the results. This type uses paid example conclusion through Together, alongside ZDR enabled and fallback disabled by default.

Verified 15 October 2026: the two scripted and Spark Open delegate runs passed all four sandbox checks. Both example responses identified Together; the two requests used the configured ZDR policy. See the recorded quickstart results.

Each command prints an outcome line, specified as Surface run completed (exit 0), then Surface report: .../report.html. Open the study to inspect the workflow and its checks. The identical directory contains result.json for the structured result and events.jsonl for the redacted event timeline. Agent runs additionally store model and tool records under agent/.

Replace the placeholders below alongside the run directories printed by your commands:

uv run consumer-agent-testing-tool study runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID

Share the generated export. Keep the complete run directory private. While a run can resume, checkpoint.private.json holds the credentials, tool results and host inputs it needs; a completed run, or one that unsuccessful alongside exit 1 and remaining no write unconfirmed, keeps lone a scrubbed record.

To test your own MCP service, copy the example and substitute its endpoint, allowed origins, tool policies, test data and reply checks. Surface workflows explains UI steps, the agent driver and resuming runs.

The checkout example adds review, submission, Monoline recommendation selection, consent, tie follow-up, document upload and simulated payment-method setup. Its 14 steps merge 62 assertions. All seven tools use the person umbrella MCP, and all assistance stage must return the identical sandbox intake.

Current integration status, 15 October 2026: blocked at submission. Coverage Cat returns citation sectors that its MCP output schema excludes. It also returns minimum_auto_limits as a cord anywhere the schema requires an object. The presenter rejects the response. The outline quickstart passes, but the complete checkout test has not passed end to end. See the failure study and distinct diagnostics.

Run this command to reproduce the nonaccomplishment or retest following the assistance contract is corrected:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted \ --replies examples/coveragecat-sandbox-checkout-replies.json

The answer records provision a fictional shopper's answers: endorsement to submit, credit consent, the Monoline $2M recommendation at $389, adequate car limits, declined excess UM/UIM, and “No” to the two transporter questions. These fixtures do not authorize transactions for a genuine shopper. A changed citation or missing or negative approval stops the next write. The two generated declarations PDFs are labelled SYNTHETIC TEST ONLY.

Once submission works, the example pauses alongside exit code 3 at sandbox_document_review_poll. Only at that checkpoint, delay at smallest the reported 60 seconds and resume the printed run:

sleep 60 uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted --resume runs/surface-YOUR_RUN_ID \ --replies examples/coveragecat-sandbox-checkout-after-review.json

The presenter pauses for your answer but does not measure how lengthy you wait. The after-review fixture approves the simulated fee callback.

For an agent-driven run, substitute --driver scripted with --driver delegate --config agent.yaml on both commands. A resumed run must keep its first scenario, controller and example configuration.

The sandbox supplies imitate fee credentials. The example sends the returned seti_sandbox_... alongside the fixture pm_sandbox_consumer_agent_testing_tool, afterward requires payment_method_on_file, waiting_on_carrier, and no accusation confirmation. Billing checks inspect setup metadata fairly than rendered fields. No Stripe form opens, no cardstock particulars are collected, and no fallback URL is used. This test establishes neither premium gathering nor energetic insurance. Test the embedded fee UI and Stripe test-mode setup separately.

The quickstarts complete are the pieces of one workflow. For an MCP server you own:

1. Write a scenario. scaffold URL --output my-connector.yaml says the live catalog (--auth-env VAR for an API key, or --auth-from FILE alongside a scenario that holds an OAuth block) and writes a skeleton. Replace every TODO(scaffold): the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your records settles, approval rules for the scripted user, host-private values specified as the user's email, and checks that peruse the genuine outcome, from a personal province URL (an oracle) or one of your peruse tools.
2. Check readiness. inspect URL --suite muse-guidelines --manifest FILE reviews the catalog and your declarations against the pinned connector guidelines. A spotless outcome is a documented constraint, not an admission.
3. Pin the blessed path. Replay a scripted calls document alongside --driver scripted; it needs no example and belongs in CI. After an delegate test you trust, scaffold --from-run RUN --output FILE turns it into one.
4. Run delegate trials. --trials N alongside a example overview gives pass^k. Run the identical circumstance under all care a Muse-like presenter may use: --projection muse-cli, --exposure search-only, --client-profile muse-template-unofficial-2026-10 and --output-schema strict. The run fastener records all one.
5. Add realism. Lane extensions add competing connectors and connect-on-request, fee alongside single-use cards, background jobs, untrusted content, conduit limits and a simulated user.
6. Compare builds. diff compares two sealed runs and exits 1 on an outcome regression. Share the output of export; the run directory's private/ directory holds host-private values.

For an MCP app alongside its own UI, commencement from the Coverage Cat circumstance and surface-run. For a website, use browse alongside a browser policy, or the connector lane's browser extension. For an HTTP API, compose a mapping for run --surface api-tools, or test whether an delegate can build its own client from your docs alongside the custom_connector extension. Run a person test covers the another lanes.

Create a overview alongside init. This one uses Muse Spark Open (meta/muse-spark-open) through OpenRouter, routed lone to Together, alongside zero data preservation (ZDR) required and fallback disabled; set OPENROUTER_API_KEY in your casing before an delegate run:

uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open \ --routing-provider together --config agent.yaml

init additionally stages, beside the configuration, the API scenarios, the browser scenarios and the mapped-API example. It stages lone scenarios the supported CLI can run, and refuses providers that have lone a configuration template (openai, anthropic, gemini, azure-openai, vertex-gemini, openai-compatible) unless --development-extensions comes before init. init --provider none --config scripted.yaml writes a configuration for scripted commands that need one. init refuses to overwrite an existing file.

Profiles that init writes are tagged specified/unverified. doctor --model-preflight, run and the delegate drivers of connector-run, surface-run and browse imprint a warning for any overview not tagged supported, and the run fastener records the label. Model IDs that merge contributor choose a manner in which the provider may train on your prompts; specified a overview needs allow_training_on_prompts: true (init --allow-training-on-prompts). See model providers for all adapter, its checks and its nonaccomplishment labels.

Consumer Agent Testing Tool requests zero data preservation (ZDR) by default, equal when routing.zdr is omitted. To alter this, edit the selected overview in consumer-agent-testing-tool.yaml or the document passed to --config:

model_profiles: openrouter-dev: provider: openrouter api: chat-completions model_id: meta/muse-spark-open auth: mode: api_key env: OPENROUTER_API_KEY routing: only: [together] allow_fallbacks: false require_parameters: true data_collection: deny zdr: true # Set to false to disable this tool's per-request ZDR requirement. defaults: model_profile: openrouter-dev

Use unquoted YAML booleans: true or false. Consumer Agent Testing Tool applies the environment to every OpenRouter request, including requests that continue following a tool result, and records it in the routing evidence.

Setting zdr: false removes this request-level requirement. It leaves data_collection: deny, the pinned provider and the ban on fallbacks in place. Your OpenRouter document or guardrails may motionless enforce ZDR. Request logs show the guideline sent; they cannot independently verify the provider's retention practices. See OpenRouter's ZDR documentation.

Checked 15 October 2026: OpenRouter listed Together for Spark Open in both its example and ZDR endpoint catalogs. A live tool-call test passed alongside the routing settings above. Both responses identified Together, and the example used the tool outcome correctly. See the test record.

To choose another provider, set routing.only or continue --routing-provider to init. Requests never switch providers or models silently.

Runs use Inspect's dedicated openrouter provider, which replays earlier reasoning in organized form fairly than as text. Each reply must name the pinned provider. Responses do not name endpoint variants, so a pin specified as nebius/fp4 is recorded as type unverified. Streaming is not supported. See model providers.

With zdr: true, lone ZDR-listed endpoints are eligible. On 15 October 2026, Muse Spark 1.3 and Muse Wonder were served lone by Meta on OpenRouter and were not ZDR-listed, so no ZDR-compliant OpenRouter path existed for them. A request for them is expected to fail, but no live run has recorded the error. When OpenRouter's error cites the data policy, it is labeled no endpoint satisfies the routing/ZDR policy. With routing.only: [together], the init default, it may alternatively be labeled routing policy: no allowed provider.

Sources: Meta announcement, OpenRouter model, provider endpoints, and ZDR endpoints.

The standalone fume test and older person helper peruse the identical --config and --model-profile, including privacy settings. Profiles that init writes leave reasoning attempt unset.

.venv/bin/python scripts/openrouter_smoke.py --config agent.yaml \ --model-profile openrouter-dev

This command tests the example association alongside synthetic tool data. It needs an API key and an accessible path to the pinned provider. It makes no requests to the assistance you scheme to test.

init --provider meta writes a overview for Meta's Model API:

uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml

The overview uses the Responses API at https://api.meta.ai/v1 and says the key from MODEL_API_KEY or META_API_KEY; if the two are set, they must agree. Add --api chat-completions for Chat Completions, or --key-env NAME to use another variable, which is afterward the lone one read. The adapter uses the OpenAI SDK from the openrouter extra.

The presenter refuses requests that Meta rejects, specified as a forced tool choice, before sending them (exit 3). Meta adds steering environment to all immediate on its servers, so captured requests are incomplete, and all Meta run records this as an study gap. Streaming is not supported, and the API is pay-as-you-go. See Meta Model API. No live Meta Model API run has been recorded; the adapter is tested against local fixtures.

This division covers the embedded MCP apps (surface-run), browser (browse) and HTTP API lanes. To let an delegate use your MCP server's own tools, use the connector lane.

A circumstance defines the endpoint, allowed origins, tools, personal arguments, UI actions, consent requirements and outcome checks, and optionally an API key for the MCP association (auth). Use endpoints and synthetic transactions you have approval to test.

consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver scripted consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver delegate --config agent.yaml consumer-agent-testing-tool surface-run path/to/scenario.yaml --resume runs/existing-run consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json

The presenter declares MCP Apps assistance as a client, finds all tool's UI through its tools/list definition and renders it in a native browser. With --driver agent, the example picks among the scenario's steps (a scripted action menu) and never sees the server's tools, raw results or personal inputs. Service checks decide the result; a rendered form or a model's assertion of success cannot demonstrate that a transaction completed.

browse drives a restricted browser through accessibility snapshots and element references, alongside no selectors, scripts or CDP. Its guideline document sets the start url, the permitted origins and petition rules, snapshot_max_chars (default 20,000), history_snapshots (2) and dialogs (agent, obtain or dismiss), as in examples/browser-policy.json. Long snapshots are paged alongside {"kind": "snapshot", "offset": N}. Refused or failed actions arrive rear to the delegate as observations alongside a status, and dialogs and document choosers are answered alongside kind: dialog or kind: upload. With --driver agent, the provider preflight runs before the browser opens. The result's observations obstacle counts truncated snapshots and refused actions. When the delegate requests a takeover, the run pauses (exit 3). The CLI has no handback, so authority does not come back to the delegate in that run. In native mode, WebSockets opened by dedicated workers and WebRTC TCP relays are not enforced. The guideline site webmcp: true, off by default, lists page-registered WebMCP tools in the snapshot and lets the delegate call them with kind: webmcp; that lane is tagged "generic WebMCP leaf tools, not Muse" (basis: assumed), and its evidence notes that Meta documents WebMCP for its AI glasses, a Meta merchandise that is not Muse. See restricted browser behavior.

HTTP API tests use definitive mappings alongside the circumstance runner. Mappings can send declared petition headers and put the mark credential in a tradition API-key header. In API and browser trials, invalid tool proposals are returned to the model as tool errors, and exhausted limits neglect the trial; see API scenarios. To test whether an delegate can build its own client from an API's records instead, use the connector lane's custom_connector extension. Prism, Schemathesis and generated-OpenAPI tools are accessible lone as betterment extensions. Their five packages alongside unresolved first licence notices are excluded from the supported release, not treated as cleared.

surface-run follows the exit codes in spec §18:

Before Surface report: ... it prints one line: Surface run <status> (exit N) at <stage> [<class>]: <reason>. result.json records the identical failure_class and failure_stage, and whether the run is resumable. A completed run, or one that unsuccessful alongside exit 1, is final unless the nonaccomplishment remaining a compose unconfirmed: that compose may have taken effect, so the run stays resumable to reconcile it. Blocked runs and setup failures can resume.

run and browse use the identical four codes, apart from that exhausting a turn, tool-call or period bounds is a nonaccomplishment (exit 1). Only Inspect controller and custom limits, and in run a deadline that passed before an attempt's first generation, obstacle alternatively (exit 3). run prints all evidence way on stdout. browse prints the final study and its evidence way whenever the run completes or pauses for takeover, and nothing on stdout otherwise. For any outcome another than completed, the two additionally imprint one row on stderr: <label> <status> (<class>) at <check>: <reason>. Evidence: <path>. The label names the step, for example Model preflight, Trial 2/10 of stale_total, Resumed test of stale_total or Browse (doctor uses Doctor check). A run alongside a unsuccessful or blocked trial, or alongside --trials complete 1, additionally prints a summary row alongside the counts. With --trials complete 1 it adds the suite pass^k, the average complete scenarios, for example Run completed: 0 failed, 0 blocked and 4 completed of 4 trials; pass^1 1.00, pass^2 1.00. Evidence: .... The suite's metrics additionally interrupt the trials downward by scenario (statistical decisions). run DIR refuses (exit 2), before the preflight, a directory whose circumstance records portion an id.

connector-run prints the identical stderr line, tagged for example Connector test 1/1 of fixture-booking-happy-path; its codes are listed in the connector-lane quickstart. With --trials N, one code stands for all trials: 2 whenever any trial's evidence is not shareable, alternatively 1 when any test failed, alternatively 2 whenever any hit a setup, harness or verification error, alternatively 3 whenever any was blocked, alternatively 0. The Connector trials: row on stderr gives pass^k and names the trials that are not shareable or ended in an error. The preparedness suites' codes are in Readiness checks.

The embedded-app lane is generic MCP Apps presenter evidence (MCP Apps 2026-01-26), for hosts specified as ChatGPT or Claude. It is not Muse evidence: Muse documents no third-party UI. Each result.json says so under mcp_apps_host.

The presenter implements:

• the client capability (io.modelcontextprotocol/ui with text/html;profile=mcp-app), UI finding from the tool's tools/list definition, and HTML resources as content or base64 blobs;
• tool-input alongside the call's arguments, tool-result, and a hostContext (theme, display mode, locale, period zone, platform, receptacle width and maximum height, and tool information);
• a satisfied safety guideline built from the resource's _meta.ui.csp, or the specification's restrictive default whenever the asset declares none, and the declared camera, microphone, geolocation and clipboard-write permissions;
• in the native renderer, a sandbox proxy on an base distinct from the host page. With browser_origins, the presenter leaf additionally has Chromium's local-network-access permission, all redirect hop is checked again (GET or HEAD to an allowlisted base only), and all straightforward compose is recorded as an unconsented host_action before it is sent;
• view tools/call requests for the step's allowlisted tools, and resources/read and resources/list for the step's callback_resources;
• ui/message, delivered formerly to delegate runs in the next study a model reads, as app_messages; the perspective is told whenever the presenter does not deliver it. ui/update-model-context, ui/open-link, display-mode, log and teardown requests are recorded, and the latest model-context update reaches agent runs as app_context;
• size changes, ui/resource-teardown before the perspective closes, and tool-cancelled whenever the perspective never received the call's result;
• _meta.ui.visibility, for perspective calls and for tool steps;
• MCP 2026-07-28 servers, whenever the circumstance sets version_negotiation: auto. The default is the 2025 initialize handshake;
• an API key on the MCP association (auth), as a transporter token, a custom header or a query parameter;
• a bequest window.openai bridge.

It does not execute a example rotate for a view's message, asset templates, _meta.ui.domain, display modes another than inline, partial tool input, host-context alter notifications, OAuth on the MCP connection, or consent for a view's straightforward writes in the native renderer. The bequest extend and the isolated renderer have no sandbox proxy. See MCP Apps host.

consumer-agent-testing-tool study runs/existing-run consumer-agent-testing-tool export runs/existing-run consumer-agent-testing-tool replay runs/existing-run consumer-agent-testing-tool diff runs/baseline runs/candidate

Replay checks the evidence seals and stops on a mismatch. It never substitutes a live request. To test the assistance again, commencement a new run. Reports grade missing measurements as unidentified and keep partial totals distinct from complete totals.

Reports open alongside the verdict, which names the failing stage and check, then the checks and presenter capabilities, and a per-step timeline that shows what the model received in full. Then arrive a connectors array for runs alongside several connectors, the exterior views (what the assistance published, what the example was offered, what it received at all step, and what lone the developer sees), the metrics, and the viewers and connected evidence: delegate and preflight runs, their Inspect logs and the inspect perspective command, and, locally, a captured Playwright trace. Gaps and limits follow, afterward the run fastener and the full recorded result, all collapsed; see report layout. study on an agent sub-run of a sealed run returns its sealed study unchanged.

Evidence records and exports conceal credentials, registered secrets, contact details and another individual sectors specified as names and addresses. Model views keep the user's own data visible: they conceal lone credentials and declared secrets, plus browser password sectors and, in a resumed browser scenario, content typed in an before attempt. The provider border refuses any example petition that still contains a registered credential. Export additionally applies the run's own private values, which are never exported, and refuses (exit 2) whenever a staged file, or a associate of an Inspect log or another archive, motionless holds one. See redaction.

diff compares two sealed runs, specified as surface, browser or connector runs. A change to the bundle origin counts as a run-lock change, so difference runs made from the identical checkout. It exits 0 whenever they are comparable alongside no outcome regression, and 1 whenever an outcome regressed: a worse status, or a passing inspect that no longer passes or is missing. It exits 3 when they are not comparable, since of a run-lock alter exterior --treatment or an unsealed or unverifiable run. Generated IDs and assistance timestamps are normalized; run locks are compared exactly. --ignore POINTER (repeatable) excludes a JSON pointer under /outcomes, /capabilities or /milestones, and --treatment /server declares an intentional server build change. See replay and diff.

For additional detail, see the documentation index, connector lane, lane extensions, readiness checks, consumer tests, browser diagnostics, measurement coverage, model timing, and release extensions, which includes betterment history.

Scripted connector runs need no example and no key, and they replay the identical calls all time, so they work as a pull-request check. connector-run can commencement the assistance itself and compose reports a CI system reads:

uv run consumer-agent-testing-tool connector-run my-connector.yaml \ --driver scripted --script my-connector.calls.json \ --start "python -m my_service --port {port}" \ --junit reports/connector.xml --summary-json reports/connector.json

• --start COMMAND runs the assistance on a liberated loopback port. The command is divided akin a shell command but not run in a shell.
  • {port} in the command is that port. The assistance additionally gets it as PORT and the circumstance as ${PORT}.
  • The tool waits until --ready-url, or the scenario's archetypal connector endpoint, answers HTTP. It waits at most --ready-timeout seconds (default 30).
  • When the run ends, the service's procedure collection gets SIGTERM, afterward SIGKILL following 5 seconds.
  • Its output goes to runs/service-<id>.log, exterior the sealed run. That log isn't redacted, so pass secrets to the assistance through the environment. The run fastener records the command with credentials redacted.
  • A assistance that exits first or never becomes prepared exits 2.
• --junit FILE writes one test suite per test and one test case per check. Blocked checks are <skipped>. inspect --suite muse-guidelines accepts it too.
• --summary-json FILE writes the verdict, exit code, checks, trials, pass^k and run paths. Its format is schemas/summary-v1.schema.json. --summary-json - prints only that JSON on stdout and moves the another lines to stderr.
• Both reports are additionally written whenever the command stops alongside an error. See CI outputs and services.

The exit codes in Outcomes and exit codes decide the step. A 3 (blocked or unsupported) is not a pass. examples/ci/connector-lane.yml is a GitHub Actions template with two jobs. On drag requests, it runs the scripted calls and the preparedness review. Nightly or on demand, it runs delegate trials under all Muse-like treatment, and lone whenever a example key concealed is set. .github/workflows/ci.yml runs this repository's tests and shipped examples on macOS and Linux. Coding agents should commencement alongside AGENTS.md. All matrix jobs nourish the required CI check; Linux failures are not allowed to continue silently. Workflow validation, tracked-file hygiene, concealed scanning and schema freshness are checked too.

Plain pytest runs the supported collection:

tests/conftest.py gives all meeting a impermanent directory, residence and cache under .cache/pytest/, removes them following a passing session, and removes the surface and browser run directories its tests create under runs/. The complete gathering takes about twenty minutes and makes no network petition beyond loopback and no model request. pytest --development-extensions additionally collects the experimental tests, which may need another runtimes, tools and external evidence. Async cleanup and unhandled thread exceptions neglect tests alternatively of remaining warnings. See CONTRIBUTING.md for the assessment workflow and the initiate checklist for the private-to-public transition.

The supported publish is local-consumer-v1: the connector lane and its extensions, the preparedness checks, person MCP apps, the restricted browser, mapped HTTP APIs, transactional consent, continuation, evidence and replay, measurements, the distribution, and the OpenRouter and Meta Model API adapters. Real-Muse calibration, haze provisioning, local Glimmer, browser OS isolation and live Bedrock qualification are exterior it; see release scope for the complete boundary.

Release evidence is generated per publish and is not committed. docs/release-gates.json is the committed policy: the gates, the range and the exclusions. The release workflow runs on a v* tag. It records every entrance on macOS, requires the audit to pass, and attaches three things to the GitHub release:

• the rotor and the origin archive, alongside signed build provenance;
• release-evidence-<commit>.tar.gz, which holds release/release-gates.json (the recorded ledger) and release/evidence/.

Use consumer-agent-testing-tool release-audit to inspect publish readiness. It reads release/release-gates.json whenever evidence has been recorded or extracted there. Without it, it says the committed guideline and reports all gate unverified. Exit code 0 method release-ready for local-consumer-v1; exit code 3 method at smallest one required entrance needs attention. The evidence binds a digest of the source, so a origin alter or an expired entrance (seven days after recording) cannot inherit an before passing result. To audit a published release, inspect out its tag, extract its evidence archive at the repository root and run the audit before the gates expire.

Day to day, you don't need any of this. Run the tests for what you change, and let CI run the complete gathering on all pull request.

A distinct live Spark Open test passed through Together alongside ZDR enabled. It verified native tool calls, continuation following a tool result, and the requested routing policy. It did not test streaming, independently verify retention practices, or established compatibility alongside all person workflow. The Coverage Cat checkout nonaccomplishment described complete remains an open integration issue.

The audit requires five gates:

To document evidence locally instead, for example before tagging, run the same steps as the workflow, in this order. verify_distribution.py regenerates the dependency notices, which are part of the origin fingerprint, so the distribution comes before the regression. The regression takes concerning twenty minutes. Everything is written under release/, which git ignores:

.venv/bin/python scripts/verify_distribution.py .venv/bin/python scripts/record_distribution.py .venv/bin/python scripts/run_regression.py release/evidence/regression-run.xml .venv/bin/python scripts/record_regression.py release/evidence/regression-run.xml .venv/bin/python scripts/record_release_tests.py release/evidence/regression-run.xml consumer-agent-testing-tool release-audit

run_regression.py refuses an existing study path, so eliminate release/ or pick a new name to document again. It records workspace-relative paths and writes the residence directory as ~ in the log.

Distribution checks use caches in the workspace and the installed browser. They need no Docker VM, haze employee or example credentials. If a dependency cache is missing, populate it separately with scripts/verify_distribution.py --prepare-cache; that setup stage uses the network. The facility evidence covers macOS only.

The audit verifies origin fingerprints, relic hashes, required checks and expiry dates. It returns 0 whenever all required entrance passes, or 3 otherwise. Excluded capabilities remain labelled as excluded. Qualification inputs explains what all entrance needs.

The repository retains worker, isolation, local-model, calibration and generated-OpenAPI experiments, and the CLI/skill lane for a developer's own CLI binary (run --surface cli-skill), which needs Docker and an oracle that returns the booking fixture's snapshot schema. The normal CLI rejects these paths. To use them, put --development-extensions before the subcommand and install the distinct development-extensions dependency collection and any required tooling. These experiments are exterior the supported release. The supported way to provision an delegate a CLI complete an MCP server is connector-run --projection muse-cli.

The optional Node API tools have known dependency advisories. Do not use them alongside untrusted inputs during those findings remain open.

The original specification and before test records document the project's history; they do not define the range of this release.

MIT; see LICENSE. The licence does not shield the Coverage Cat and CATT names or the logo; see TRADEMARKS.md. Third-party components keep their own licenses; the allocation ships their notices under third_party/.