Connect an output check

Goal: Prepare or reuse an output check, connect it to Millwork, select it for one task, and read its result. The API calls an output check a verifier. The full walkthrough is complete when: your handler accepts and rejects the expected examples, Millwork can reach it, and a live run’s receipt names your saved check and records its verdict and available named anchors. If you only need to connect the check, stop after Millwork confirms access and a compatible response. Your check returns two signals: whether the output meets your correctness rules (is_correct) and a separate quality score (quality_score). Millwork records these signals and uses them with the follow-up actions you allow. You own the rules; Millwork does not independently verify that they are right for your product. You can start with a compatible HTTPS endpoint you already operate, or scaffold Millwork’s maintained adapter and deployment recipe. A third-party evaluator may need that adapter to translate its request and result shapes.

Before you begin

To prepare and test an adapter locally, use Node.js 20 or 22 with Bash or Zsh. Install the exact CLI release used by this walkthrough:

npm install --global @millwork/solver@0.1.20

Then check its version and available commands:

millwork --version --json
millwork --help

The help output must include verifier init, verifier connect, and verifier test --local. If any is absent, stop; that installed CLI does not support this guide. This local check does not require an account key. Keep that version for every CLI command on this page. Do not substitute a checkout build or an unpinned @latest; either can expose different features. To connect an endpoint, use an approved organization. You need an HTTPS route you can deploy, or an existing compatible endpoint. The connection examples also use jq. A live model run additionally needs a saved model, credit, quota, and your approval. You can stop after connecting the check. Choose access before you deploy:

Access	Choose it when	What stays private
Public	The endpoint needs no credential and every candidate contains only public or test data.	No adapter key exists. Do not send tenant-internal data to this route.
Protected	The endpoint requires a dedicated adapter key.	Enter the key through a hidden terminal prompt or customer-controlled automation. Never put it in a URL, shell history, support message, or assistant conversation.

Keep any evaluator-provider key inside your app. It is separate from the adapter key Millwork sends to your endpoint. Choose the route that matches your task:

Prepare a new endpoint, then connect it.
Connect an existing compatible endpoint.
Use a saved check on a live run.

This diagram shows the successful path and the point where setup can stop.

What Millwork sends

Every call is one HTTPS POST with a candidate field. For the model run in this guide, the candidate is the model’s response as a string. Asking for JSON does not turn that string into an object:

{
  "candidate": "{\"extracted_fields\":{\"price\":19.99},\"summary\":\"A used desk lamp in good condition.\"}"
}

The maintained adapter parses this string before checking its fields. Invalid JSON or an unexpected shape fails the check. Other saved options, such as an agent endpoint, can return a different candidate shape; adapt the check to the output you actually send. The request includes no run ID or check ID. Keep trusted references and acceptance rules in your handler’s configuration. Treat candidate text as data, never as code to execute. Registration and POST /v1/verifiers/{verifierId}/test send this reserved candidate:

{
  "candidate": {
    "solverapi_probe": "registration_preflight"
  }
}

What your handler must return

Return 200 with JSON. is_correct and quality_score are required on every success response, including the probe.

{
  "is_correct": true,
  "quality_score": 0.62,
  "anchor_results": {
    "has_required_field_price": true,
    "price_is_positive_number": true
  }
}

Field	Rule
`is_correct`	Boolean. Your deterministic correctness rules. No language-model judge in this path.
`quality_score`	Number in `0` to `1`. Your quality score. It never overrides `is_correct`.
`anchor_results`	Optional object of booleans. One named assertion per key. Do not put customer values in the key names; those names can reach the receipt.
`named_metrics`	Optional object of numbers. Millwork may accept it on the response. It does not appear on receipts.

A quoted "true", a quality_score outside 0 to 1, or a non-boolean anchor is a contract failure, not a failed check.

Transport facts

These are constraints of the call, not defaults you can override.

Fact	What your handler must do
Probe budget is 3 seconds	Answer the reserved probe without calling a slow backend.
Runtime budget is 10 seconds	Finish a live check in 10 seconds. Millwork does not retry and does not extend the budget.
Request is candidate-only	Do not wait for a correlation id that will not arrive.
POST may be chunked	Accept a chunked body. Millwork does not send `Content-Length`. Some serverless gateways reject that.
`named_metrics` stay off receipts	Put durable evidence in `is_correct`, `quality_score`, and `anchor_results`.
Contract validation does not gate `ready`	A reachable probe can still fail the result schema. Read `preflight.contract` (or `probe.contract` on test).

The URL must be https on a public address. Do not put a secret in the URL. Millwork does not follow redirects.

Hard rules and quality rules

Keep the two signals separate.

Hard (is_correct): schema checks, exact matches, checksums, required fields, ordered steps, other tenant-owned deterministic assertions.
Quality (quality_score): return a number from 0 to 1 on every successful response. A separate model judge is optional; the score is not. Calculate it with the scoring function in your generated check.
A high quality score never turns a failed assertion into is_correct: true.
A low quality score never turns a passed assertion into is_correct: false.
Changing only the judge or quality function must not flip is_correct.

Test the verdict and score separately

Try these cases through the same handler you will register:

A candidate with a missing or non-positive price and a long summary must have is_correct: false, even if its quality score is high.
A candidate with a positive numeric price and a short summary can have is_correct: true and a low quality score.
Change only the quality function. The correctness answer must stay the same.

These tests check how your handler combines the signals. They do not prove that its rules cover every mistake your product needs to catch.

Reserved probe handling

Treat only the exact reserved object as a probe:

function isReservedProbe(candidate) {
  return (
    candidate !== null
    && typeof candidate === "object"
    && !Array.isArray(candidate)
    && Object.keys(candidate).length === 1
    && candidate.solverapi_probe === "registration_preflight"
  );
}

If it is the reserved probe, return a valid negative result quickly: is_correct: false, quality_score: 0. Do not call your real check. A live candidate equal to the marker must not pass a production gate. Registration and test still accept this body; ready does not require is_correct: true.
If solverapi_probe appears in any other shape, fail closed. Do not take the probe shortcut. Never return a production pass because the marker was present. Returning is_correct: false with quality_score: 0 is the closed path.
The candidate-only request cannot prove who called you. A marker must never grant a live pass.

Prepare the endpoint

If you already operate a compatible endpoint, keep its HTTPS URL and continue to Deploy and connect the check. Skip the scaffold and deployment instructions below. Otherwise, scaffold the maintained protected adapter and deployment recipe in an empty directory:

millwork verifier init output-check --json
cd output-check

The generated directory contains the adapter, a compatibility kit, an example check, an existing-app integration, and DEPLOYMENT_RECIPE.md. Edit listing-example-check.mjs so its hard rules and quality function match your task. Put your task’s rules in that file; keep the adapter’s request and probe handling. Test the public shape locally:

millwork verifier test --local \
  --check listing-example-check.mjs \
  --access public \
  --json

For a protected endpoint, test the adapter’s authentication, overlap, and retirement cases instead:

millwork verifier test --local \
  --check listing-example-check.mjs \
  --access authenticated \
  --json

Expected result: headline is usable and every labelled case passes. Intentional failure cases count as passed when the adapter refuses them in the expected way. If a case fails, correct the rule or expected result and run the same local test again. Next, deploy the tested check and test its HTTPS endpoint.

Deploy and connect the check

Before connecting, configure your organization API key using the private setup instructions. Never paste the key into a support message or agent conversation. Then confirm account access:

millwork verifier list --json

The generated DEPLOYMENT_RECIPE.md is for a protected route in a Node app you operate. From the generated directory, follow that recipe using paths relative to the directory. When adding the listener to your app, import createExistingNodeAppListener from ./existing-node-app.mjs and import runHardCheck and scoreQuality from ./listing-example-check.mjs. Pass those two functions to the listener so the deployed route uses the check you tested. Deploy through your app’s normal release path, then test the protected HTTPS route as the recipe directs. If you choose public access, start with an already-deployed compatible endpoint that accepts requests without a credential; the maintained protected recipe does not create a public route. Millwork does not deploy the endpoint for you and does not follow redirects. Keep the route available. When you change its scoring logic, update the registered version and test it again.

export VERIFIER_URL="<your deployed HTTPS endpoint>"

Connect a public endpoint

Use this path only when the endpoint needs no authentication and candidates contain public or test data:

millwork verifier connect \
  --endpoint "$VERIFIER_URL" \
  --access public \
  --name "Listing extraction check" \
  --version "1.0.0" \
  --connect-only

--connect-only registers and tests the check without starting a model run. The command returns a saved check ID. Keep it as VERIFIER_ID; registration does not automatically select it for later tasks. For a script or coding assistant, keep the metadata on verifier connect, add --json, and save its structured result:

if ! millwork verifier connect \
  --endpoint "$VERIFIER_URL" \
  --access public \
  --name "Listing extraction check" \
  --version "1.0.0" \
  --connect-only \
  --json > verifier-connect.json; then
  echo "Connection failed. Read verifier-connect.json before continuing." >&2
  exit 1
fi

jq -e '.headline == "ready" and .probe.contract.validated == true' \
  verifier-connect.json >/dev/null || exit 1
VERIFIER_ID="$(jq -er '.verifier_id | strings | select(length > 0)' verifier-connect.json)" || exit 1
export VERIFIER_ID

A failed connection can still return an ID. Keep it for inspection and repair, but do not use it for a run until the headline is ready and the response contract is validated. The execution MCP can list, show, connect, and retest a public endpoint when its installed tool list includes the verifier tools. It requires the same URL, name, version, access choice, and a caller-owned request key. Its connect tool does not start a model run and never accepts an adapter secret. For protected access, inspect the installed MCP tool list before choosing the continuation. When solver_start_verifier_connection, solver_inspect_verifier_connection, solver_continue_verifier_connection, and solver_disconnect_verifier_connection are present, register with access: "managed", start the protected intent, and pause at action_required. Give the returned continue_url only to the intended person or configured customer automation. After private entry, continue the same operation_key; an authenticated connection read is the completion proof. No tool accepts the endpoint key. If any lifecycle tool is absent, use the CLI or dashboard flow below and do not invent a tool name or handwritten HTTP call.

Connect a protected endpoint

At the endpoint, issue a dedicated adapter key and configure the deployed app to accept it. Then choose when Millwork should stop using its stored copy: Set the Millwork app address for this terminal:

export CUSTOMER_APP_ORIGIN="https://app.getmillwork.dev"

For another deployment, use the app address supplied by its administrator. This is the Millwork app address, not your output-check endpoint.

millwork verifier connect \
  --endpoint "$VERIFIER_URL" \
  --access managed \
  --name "Listing extraction check" \
  --version "1.0.0" \
  --stop-days 90 \
  --connect-only

Enter the key only in the hidden prompt. Over SSH, use an interactive TTY and run the same command; the prompt does not echo the key. Retain the saved check ID printed when the command completes. Configured customer automation may provide VERIFIER_CONNECTION_SECRET in the command’s host environment. The assistant must not read, print, create, or change that environment value. A secret is never a command argument or MCP tool input. Capture automated JSON in an owner-only local file and confirm its connection status is active and verifier_id is nonempty before retaining the ID. Without a private interactive terminal or configured automation secret, the command returns state: "action_required" with verifier_id, continue_url, intent_id, and expires_at. Sign in to Millwork in your normal browser using the same organization and an account that can manage checks. Open the private continue_url to enter the key. Then finish that saved connection:

millwork verifier continue \
  --verifier-id "<verifier_id from action_required>" \
  --intent-id "<intent_id from action_required>" \
  --json

Use the same organization API key for the continuation and any retry. Human terminal usage may omit --json; an assistant must keep JSON and capture it in the owner-only local surface. Confirm connection.status is active and verifier_id matches the saved ID. Keep the verifier and intent IDs together, keep continue_url on the private handoff surface, and do not expose the key-entry session to the assistant. After the saved protected connection is active, retain its check ID:

export VERIFIER_ID="<saved check ID from this connection>"

A pending ID identifies an attempt; it is not evidence that the connection is active. The protected connection response does not use the public connection’s headline and probe fields. Run a fresh remote test to confirm the protected endpoint, and repeat it after changing the endpoint or its access:

export TEST_REQUEST_KEY="verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')"
millwork verifier test \
  --verifier-id "$VERIFIER_ID" \
  --idempotency-key "$TEST_REQUEST_KEY" \
  --json

Expected result: the test has headline ready, probe.contract.validated is true, and its verifier_id matches VERIFIER_ID. The reserved probe may return is_correct: false; that negative verdict is intentional and does not make a compatible endpoint unavailable. Use a new request key for each new observation. If that test’s response is lost, retry it with the same key. Do not assume a later model run retests the connection for you. If you only need to connect the check, stop after this test confirms access and a compatible response.

What the recorded hash covers

The hash covers the registered definition: display name, version, URL, input data classes, and scoring shape. It does not fingerprint deployed code, models, prompts, datasets, or dependencies behind a stable URL. Changing scoring logic without bumping version (or another hashed field) does not necessarily produce a new hash on later receipts. Versioning is your obligation. Repeated probes can falsify a determinism claim. They cannot prove the absence of a language model, a cache, or mutable external state. A queue-existence or URL-resolvability check depends on changing external state. Determinism, correctness coverage, independence, and availability are four different properties. Observed repeatability is not independent verification.

Describe your check’s correctness rules

The API accepts an optional correctness_declaration with { "method": "deterministic" } when you register or update a check. This records your organization’s statement; Millwork does not independently verify it. A declaration-only update changes the revision, not the definition hash. New evaluations record the declaration in effect at that time. A receipt can show “Tenant declared deterministic correctness; not independently verified.” Older receipts do not gain a declaration when you add one later. See the API reference for registration, updates, and reads.

The endpoint test fails

What you see	What it means	What to do
`status` `degraded`, reason `dns_resolution_failed` or unreachable	Millwork could not reach the public HTTPS URL	Confirm the hostname, TLS, and that the address is public. Retest.
HTTP `401` or `403` on a credentialed probe	Authentication failed	Correct the adapter credential your handler expects. Retest.
HTTP `2xx` and `contract.validated` false	Reachable, wrong body	Fix `is_correct`, `quality_score`, or `anchor_results`. Retest.
Probe exceeds 3 seconds	The reserved shortcut did not run, or a backend ran during probe	Short-circuit the exact reserved candidate. Retest.
The output check takes longer than 10 seconds	The check timed out	Make the handler respond within 10 seconds, then test it again. Millwork does not retry the check.

After each correction, run the remote test again:

export TEST_REQUEST_KEY="verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')"
millwork verifier test \
  --verifier-id "$VERIFIER_ID" \
  --idempotency-key "$TEST_REQUEST_KEY" \
  --json

Use a new request key after each correction. Reuse a key only to recover that exact test. Do not assume a later run will retest for you.

Use the check on a live run

A live run charges a platform fee when accepted and may incur separate model usage. It needs credit and quota. Review model access and billing before continuing. The handler in this example adds no model call of its own. You need a saved model (the API calls it an arm). If you do not have one, open Run a model, choose and save a model, then return before submitting its example run. If CLI setup selected your current model, read Preset: and Managed arm: from the same current application:

millwork tenant show

Keep those two values. The command below names the model, preset, check, and task explicitly. Omitting --verifier-id would use the basic output-presence check, which does not judge whether the answer is correct.

export ARM_ID="<current model ID>"
export PRESET="<preset from tenant show>"

millwork run \
  --preset "$PRESET" \
  --arm-id "$ARM_ID" \
  --verifier-id "$VERIFIER_ID" \
  --objective 'Return only JSON: {"extracted_fields":{"price":number},"summary":string}. Extract the price from "Used desk lamp, good condition. Price: 19.99." and write a brief summary.'

If you have no proved preset, use the saved model’s arm_id and set the limits explicitly. Choose positive values that fit this task; the guide does not choose a spending limit for you.

export ARM_ID="<saved model ID>"
export MAX_COST_USD="<your positive model budget in USD>"
export MAX_RUNTIME_S="<your positive whole-second limit>"
export DATA_CLASS="<public, sandbox, or tenant_internal>"

millwork run \
  --arm-id "$ARM_ID" \
  --verifier-id "$VERIFIER_ID" \
  --max-cost-usd "$MAX_COST_USD" \
  --max-runtime-s "$MAX_RUNTIME_S" \
  --data-class "$DATA_CLASS" \
  --objective 'Return only JSON: {"extracted_fields":{"price":number},"summary":string}. Extract the price from "Used desk lamp, good condition. Price: 19.99." and write a brief summary.'

The preview shows these limits and the platform fee before you approve the run.

The host approves the run, not a prompt

Review the preview before anything is charged. It shows the model, provider, payer, data classes, runtime, platform fee, model budget, combined allowance, request hash, and replay key. A paid run is approved by the administrator of the machine you run on. There is no approval prompt to answer. --yes cannot approve a paid run, an interactive terminal cannot approve one, and neither can a coding assistant. Every paid run needs a one-run approval that your host writes for that exact request, through an approval channel the host configured. There is no standing setting that approves runs ahead of time on your behalf. In a script or coding assistant, add --json. When the run stops for that approval, next_action.type is obtain_host_one_run_approval, and the response carries the three things you need:

host_approval.approval_file is the file to create.
host_approval.document is what goes in it.
replay_command is the command to run once the file is there.

Only your host administrator can create that file. When it exists, run replay_command exactly as printed. It carries the replay key this approval covers; your original command would create a new key and ask for a different run. If the machine has no run-authorization boundary configured, the run refuses before anything is sent and names the check that failed. Ask your host administrator to configure it as the SDK’s README describes. If next_action.type is refresh_billing_configuration, update or refresh the reported billing configuration first. That action does not approve a run. Do not create a different request while either action is unresolved. If the submission outcome is unknown after an attempted submission, run the CLI’s printed replay_command unchanged. It carries the same replay identity and recovers that run. This is recovery, not an approval handoff; keep --json when it is present. A new command without that identity requests new paid work. The interactive command waits for a final state and prints the answer, run status, receipt ID, model usage, and platform fee. To inspect it again without submitting work, use the retained run ID:

millwork run --execution-id "<run ID>" --json

Expected result: the receipt names VERIFIER_ID, records a boolean is_correct, a separate quality_score, and any anchor_results returned by the check. CLI JSON keeps these records under execution, result, and receipt; retain the run ID and inspect those wrappers from the same run. A negative verdict means the output did not meet your rules even when its quality score is high. Read Results and receipts for the exact fields and dashboard labels.

The receipt has no verdict

A missing verdict or NOT CHECKED is not a rejection. Read the run status and the receipt’s verification_checks to see whether your check was attempted. Confirm that the receipt’s verifier ID matches VERIFIER_ID. A queued or running task is still in progress, and a task can end before its check is attempted. If a customer check records technical_failure, fix its reported failure and run a fresh endpoint test before deciding whether to submit another paid task. Do not replace a working credential based only on an absent verdict.

A protected connection needs recovery

Inspect and retest before changing a connection:

millwork verifier show --verifier-id "$VERIFIER_ID" --json
export TEST_REQUEST_KEY="verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')"
millwork verifier test \
  --verifier-id "$VERIFIER_ID" \
  --idempotency-key "$TEST_REQUEST_KEY" \
  --json

To replace an active key, first configure your endpoint to accept the current and new keys. Then run:

millwork verifier replace \
  --verifier-id "$VERIFIER_ID" \
  --stop-days 90

Enter the new key privately. Millwork tests it and switches only after a pass. A failed replacement leaves the previous key selected; it does not make an expired or revoked key active. After a successful switch, remove the old key at your endpoint and confirm that endpoint rejects it. If setup returned a verifier ID and intent ID for private entry, finish it with the verifier continue --verifier-id ... --intent-id ... command above. For replacement or restore, use the exact continuation the CLI printed. For an unknown or pending outcome, preserve its operation key and handle and use the same organization API key. If initial terminal setup was interrupted without those references, inspect the existing connection before changing it again. Do not create another connection to discover what happened. Check the connection’s current status before using the new key. Disconnecting stops Millwork from sending the key but does not revoke it at your endpoint:

millwork verifier disconnect --verifier-id "$VERIFIER_ID" --yes

Remove or revoke the key at your endpoint too, then confirm it is rejected. To restore a disconnected or stopped connection, issue a new endpoint key and run:

millwork verifier restore \
  --verifier-id "$VERIFIER_ID" \
  --stop-days 90

Millwork never reuses the disconnected or expired key. An old or expired private entry cannot restore access; start a fresh restore and use its printed continuation.

Results and receipts

Read the run’s output and understand the evidence on its receipt.

Errors and retries

Recover a failed request without starting work twice.

Get started

Provide models

Build

Build safely

SDK and MCP

Connect an output check

Before you begin

What Millwork sends

What your handler must return

Transport facts

Hard rules and quality rules

Test the verdict and score separately

Reserved probe handling

Prepare the endpoint

Deploy and connect the check

Connect a public endpoint

Connect a protected endpoint

What the recorded hash covers

Describe your check’s correctness rules

The endpoint test fails

Use the check on a live run

The host approves the run, not a prompt

The receipt has no verdict

A protected connection needs recovery

Results and receipts

Errors and retries

​Before you begin

​What Millwork sends

​What your handler must return

​Transport facts

​Hard rules and quality rules

​Test the verdict and score separately

​Reserved probe handling

​Prepare the endpoint

​Deploy and connect the check

​Connect a public endpoint

​Connect a protected endpoint

​What the recorded hash covers

​Describe your check’s correctness rules

​The endpoint test fails

​Use the check on a live run

​The host approves the run, not a prompt

​The receipt has no verdict

​A protected connection needs recovery

Results and receipts

Errors and retries

Before you begin

What Millwork sends

What your handler must return

Transport facts

Hard rules and quality rules

Test the verdict and score separately

Reserved probe handling

Prepare the endpoint

Deploy and connect the check

Connect a public endpoint

Connect a protected endpoint

What the recorded hash covers

Describe your check’s correctness rules

The endpoint test fails

Use the check on a live run

The host approves the run, not a prompt

The receipt has no verdict

A protected connection needs recovery