> ## Documentation Index > Fetch the complete documentation index at: https://docs.getmillwork.dev/llms.txt > Use this file to discover all available pages before exploring further. # Connect an output check > Connect your own output check, use it on a live run, and confirm its result on the receipt. **Goal:** Connect an **output check** you already run so Millwork can apply it to a model's response. The API calls an output check a verifier. **You are done when:** your handler accepts and rejects the expected examples, and a live run's receipt names your saved check and records its result. Your check returns two signals: whether the output meets your correctness rules (`is_correct`) and a separate quality score (`quality_score`). Millwork records these signals and uses them with the follow-up actions you allow. You own the rules; Millwork does not independently verify that they are right for your product. To use a third-party evaluator, check that its endpoint accepts and returns the formats below. You may need a handler that translates between them. ## Before you begin * Your organization is approved for the private preview. * You have Node.js 20 or later, Bash or Zsh, `curl`, and `jq`. * You can add a public HTTPS route to an app you run. * You will use public information or examples created for testing. Configure your organization API key using the [private terminal instructions](/get-started/tenant-start#private-terminal-fallback), or [load a saved key](/get-started/tenant-start#load-a-saved-key). Those commands set `SOLVERAPI_API_KEY` in your terminal. Signing in through the CLI's browser window alone does not set that variable for `curl`. Never paste a key into a support message or agent conversation. Use the same terminal for the rest of this guide: ```bash theme={null} : "${SOLVERAPI_API_KEY:?Load your Millwork key in this terminal first.}" export MILLWORK_API_KEY="$SOLVERAPI_API_KEY" export MILLWORK_API_URL="https://api.getmillwork.dev/v1" ``` The example handler has no authentication and makes no calls to a model or other paid service. Use it only with public or test data. To protect an endpoint, [ask support](/help/contact) to help configure the credential Millwork will send. Do not send the credential in that message. Keep any evaluator-provider key in your app; it is not the credential Millwork uses to call your handler. The final step submits a live model run and needs credit and quota. You can build, register and test the handler before choosing to run that paid step. ## What Millwork sends Every call is one HTTPS POST with a `candidate` field. For the model run in this guide, the candidate is the model's response **as a string**. Asking for JSON does not turn that string into an object: ```json theme={null} { "candidate": "{\"extracted_fields\":{\"price\":19.99},\"summary\":\"A used desk lamp in good condition.\"}" } ``` The handler below parses this string before checking its fields. Invalid JSON or an unexpected shape fails the check. Other saved options, such as an agent endpoint, can return a different candidate shape; adapt the handler to the output you actually send. The request includes no run ID or check ID. Keep trusted references and acceptance rules in your handler's configuration. Treat candidate text as data, never as code to execute. Registration and `POST /v1/verifiers/{verifierId}/test` send this reserved candidate: ```json theme={null} { "candidate": { "solverapi_probe": "registration_preflight" } } ``` ## What your handler must return Return `200` with JSON. `is_correct` and `quality_score` are required on every success response, including the probe. ```json theme={null} { "is_correct": true, "quality_score": 0.62, "anchor_results": { "has_required_field_price": true, "price_is_positive_number": true } } ``` | Field | Rule | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `is_correct` | Boolean. Your deterministic correctness rules. No language-model judge in this path. | | `quality_score` | Number in `0` to `1`. Your quality score. It never overrides `is_correct`. | | `anchor_results` | Optional object of booleans. One named assertion per key. Do not put customer values in the key names; those names can reach the receipt. | | `named_metrics` | Optional object of numbers. Millwork may accept it on the response. It does not appear on receipts. | A quoted `"true"`, a `quality_score` outside `0` to `1`, or a non-boolean anchor is a contract failure, not a failed check. ## Transport facts These are constraints of the call, not defaults you can override. | Fact | What your handler must do | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | Probe budget is 3 seconds | Answer the reserved probe without calling a slow backend. | | Runtime budget is 10 seconds | Finish a live check in 10 seconds. Millwork does not retry and does not extend the budget. | | Request is candidate-only | Do not wait for a correlation id that will not arrive. | | POST may be chunked | Accept a chunked body. Millwork does not send `Content-Length`. Some serverless gateways reject that. | | `named_metrics` stay off receipts | Put durable evidence in `is_correct`, `quality_score`, and `anchor_results`. | | Contract validation does not gate `ready` | A reachable probe can still fail the result schema. Read `preflight.contract` (or `probe.contract` on test). | The URL must be `https` on a public address. Do not put a secret in the URL. Millwork does not follow redirects. ## Hard rules and quality rules Keep the two signals separate. * **Hard (`is_correct`):** schema checks, exact matches, checksums, required fields, ordered steps, other tenant-owned deterministic assertions. * **Quality (`quality_score`):** return a number from `0` to `1` on every successful response. A separate model judge is optional; the score is not. Calculate it with your scoring function, such as the example below. * A high quality score never turns a failed assertion into `is_correct: true`. * A low quality score never turns a passed assertion into `is_correct: false`. * Changing only the judge or quality function must not flip `is_correct`. ## Test the verdict and score separately Try these cases through the same handler you will register: * A candidate with a missing or incorrect price and a long summary must have `is_correct: false`, even if its quality score is high. * A candidate with the expected price and a short summary can have `is_correct: true` and a low quality score. * Change only the quality function. The correctness answer must stay the same. These tests check how your handler combines the signals. They do not prove that its rules cover every mistake your product needs to catch. ## Reserved probe handling Treat only the exact reserved object as a probe: ```javascript theme={null} function isReservedProbe(candidate) { return ( candidate !== null && typeof candidate === "object" && !Array.isArray(candidate) && Object.keys(candidate).length === 1 && candidate.solverapi_probe === "registration_preflight" ); } ``` * If it is the reserved probe, return a valid **negative** result quickly: `is_correct: false`, `quality_score: 0`. Do not call your real check. A live candidate equal to the marker must not pass a production gate. Registration and test still accept this body; `ready` does not require `is_correct: true`. * If `solverapi_probe` appears in any other shape, **fail closed**. Do not take the probe shortcut. Never return a production pass because the marker was present. Returning `is_correct: false` with `quality_score: 0` is the closed path. * The candidate-only request cannot prove who called you. A marker must never grant a live pass. ## Build the handler Save this example as `output-check.mjs` and run it with `node output-check.mjs`. It checks that a listing's price matches the trusted price `19.99` configured in the handler. Its quality score measures summary length, capped at 200 characters; it does not judge the summary's accuracy. Replace these example rules with the checks and scoring function you use for your own task. ```javascript theme={null} import { createServer } from "node:http"; function isReservedProbe(candidate) { return ( candidate !== null && typeof candidate === "object" && !Array.isArray(candidate) && Object.keys(candidate).length === 1 && candidate.solverapi_probe === "registration_preflight" ); } function containsProbeMarker(value) { if (value === null || typeof value !== "object") return false; if (Object.prototype.hasOwnProperty.call(value, "solverapi_probe")) return true; return Object.values(value).some(containsProbeMarker); } function runHardCheck(candidate) { const price = candidate?.extracted_fields?.price; const anchors = { has_required_field_price: typeof price === "number", price_is_positive_number: typeof price === "number" && price > 0, price_matches_listing: price === 19.99, }; return { is_correct: Object.values(anchors).every(Boolean), anchor_results: anchors }; } function scoreQuality(candidate) { const text = typeof candidate.summary === "string" ? candidate.summary : ""; return Math.min(1, text.length / 200); } async function readJson(request) { const chunks = []; for await (const chunk of request) chunks.push(chunk); return JSON.parse(Buffer.concat(chunks).toString("utf8")); } const server = createServer(async (request, response) => { const send = (status, body) => { response.writeHead(status, { "content-type": "application/json" }); response.end(JSON.stringify(body)); }; if (request.method !== "POST") { send(405, { is_correct: false, quality_score: 0 }); return; } let payload; try { payload = await readJson(request); } catch { send(400, { is_correct: false, quality_score: 0 }); return; } let candidate = payload?.candidate; if (isReservedProbe(candidate)) { send(200, { is_correct: false, quality_score: 0 }); return; } if (containsProbeMarker(candidate)) { send(200, { is_correct: false, quality_score: 0 }); return; } // Model output is text, even when it contains JSON. if (typeof candidate === "string") { try { candidate = JSON.parse(candidate); } catch { send(200, { is_correct: false, quality_score: 0 }); return; } } if (candidate === null || typeof candidate !== "object" || Array.isArray(candidate) || containsProbeMarker(candidate)) { send(200, { is_correct: false, quality_score: 0 }); return; } const hard = runHardCheck(candidate); send(200, { is_correct: hard.is_correct, quality_score: scoreQuality(candidate), anchor_results: hard.anchor_results, }); }); server.listen(8080); ``` Serve this route over public HTTPS. Terminate TLS in front of this process if your app already does. Register the public `https` URL, not `http://127.0.0.1`. ## Deploy and register the handler Deploy the handler in your app and confirm that its public HTTPS URL is reachable. Millwork does not deploy it for you. Keep the route available; when you change its scoring logic, update the registered `version` and test it again. If its authentication changes, arrange the new credential binding with support before using it for another run. Replace this example URL with your handler's URL. A **request key** is the value sent in the `Idempotency-Key` header so retrying the same request does not repeat the work. Generate one for registration and keep it with the body: ```bash theme={null} export VERIFIER_URL="https://your-app.example/output-check" export REGISTER_REQUEST_KEY="check-register-$(node -e 'process.stdout.write(crypto.randomUUID())')" jq -n --arg url "$VERIFIER_URL" '{ display_name: "Listing extraction check", version: "1.0", kind: "endpoint", endpoint: { url: $url, auth_ref: "" }, input_data_classes: ["public"], scoring: { correctness: "boolean_anchors", quality: "scalar_0_1" } }' > verifier-request.json if ! curl --fail-with-body \ --request POST \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $REGISTER_REQUEST_KEY" \ --data @verifier-request.json \ --output verifier-response.json \ "$MILLWORK_API_URL/verifiers"; then echo "Registration failed. Read verifier-response.json before continuing." >&2 exit 1 fi VERIFIER_ID="$(jq -er '.verifier_id | strings | select(length > 0)' verifier-response.json)" || exit 1 export VERIFIER_ID ``` **Expected result:** `201` with a nonempty `verifier_id`, `hash`, `status`, and `preflight`. `auth_ref: ""` means this endpoint needs no authentication. Keep `VERIFIER_ID`; the live request below uses it. If the response was lost, resend the same body with the same `REGISTER_REQUEST_KEY`. Reusing that key returns the stored response, including a stored error; after correcting a rejected request, use a new key. Test the saved check with a fresh request key. Reuse this key only to recover this particular test request, not to make a new observation after fixing the handler: ```bash theme={null} export TEST_REQUEST_KEY="check-test-$(node -e 'process.stdout.write(crypto.randomUUID())')" if ! curl --fail-with-body \ --request POST \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --header "Idempotency-Key: $TEST_REQUEST_KEY" \ --output check-test.json \ "$MILLWORK_API_URL/verifiers/$VERIFIER_ID/test"; then echo "The test request failed. Read check-test.json before continuing." >&2 exit 1 fi jq -e '.status == "ready" and .probe.outcome == "reachable" and .probe.contract.validated == true' check-test.json >/dev/null || exit 1 ``` **Expected result:** `ready`, `reachable`, and a validated response. The example's reserved probe returns `is_correct: false` and `quality_score: 0`; that negative answer is intentional. `ready` means the probe had no reachability or authentication failure. It does not certify your correctness rules, and it does not by itself mean the response matched the required schema. ## What the recorded hash covers The hash covers the registered definition: display name, version, URL, input data classes, and scoring shape. It does not fingerprint deployed code, models, prompts, datasets, or dependencies behind a stable URL. Changing scoring logic without bumping `version` (or another hashed field) does **not** necessarily produce a new hash on later receipts. Versioning is your obligation. Repeated probes can falsify a determinism claim. They cannot prove the absence of a language model, a cache, or mutable external state. A queue-existence or URL-resolvability check depends on changing external state. Determinism, correctness coverage, independence, and availability are four different properties. Observed repeatability is not independent verification. ## Describe your check's correctness rules The API accepts an optional `correctness_declaration` with `{ "method": "deterministic" }` when you register or update a check. This records your organization's statement; Millwork does not independently verify it. A declaration-only update changes the revision, not the definition hash. New evaluations record the declaration in effect at that time. A receipt can show "Tenant declared deterministic correctness; not independently verified." Older receipts do not gain a declaration when you add one later. See the [API reference](/api-reference/overview) for registration, updates, and reads. ## If registration or test fails | What you see | What it means | What to do | | ------------------------------------------------------------------ | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `status` `degraded`, reason `dns_resolution_failed` or unreachable | Millwork could not reach the public HTTPS URL | Confirm the hostname, TLS, and that the address is public. Retest. | | HTTP `401` or `403` on a credentialed probe | Authentication failed | Correct the adapter credential your handler expects. Retest. | | HTTP `2xx` and `contract.validated` false | Reachable, wrong body | Fix `is_correct`, `quality_score`, or `anchor_results`. Retest. | | Probe exceeds 3 seconds | The reserved shortcut did not run, or a backend ran during probe | Short-circuit the exact reserved candidate. Retest. | | The output check takes longer than 10 seconds | The check timed out | Make the handler respond within 10 seconds, then test it again. Millwork does not retry the check. | After each correction, call `POST /v1/verifiers/{verifierId}/test` again. Do not assume a later run will retest for you. ## Use the check on a live run A live run charges a **platform fee when accepted** and may incur separate model usage. It needs credit and quota. Review [model access and billing](/concepts/access) before continuing. The handler in this example adds no model call of its own. You need a **saved model** (the API calls it an arm). If you do not have one, open [Run a model](/get-started/builder) and complete **Choose a model from your catalog** and **Save the model to your organization**. Return here **before** its **Submit the run** step. Keep its `ARM_ID` in this terminal. You do not need to run its example task. Check the saved model and both IDs before preparing the paid request: ```bash theme={null} : "${VERIFIER_ID:?Register the output check first.}" : "${ARM_ID:?Choose and save a model, then return here with ARM_ID.}" if ! curl --fail-with-body \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --output saved-model.json \ "$MILLWORK_API_URL/arms/$ARM_ID"; then echo "Could not read the saved model. Read saved-model.json." >&2 exit 1 fi jq -e --arg arm "$ARM_ID" '.arm_id == $arm and .status == "ready"' saved-model.json >/dev/null || exit 1 ``` The task below asks for the JSON text the handler expects. It explicitly attaches your `VERIFIER_ID`; omitting that field would use a basic output-presence check instead. `on_eval: []` requests no repair or fallback action after the check. The model-usage budget stops further calls after recorded usage reaches it; a call already started can finish above it. The platform fee is separate. ```bash theme={null} export RUN_REQUEST_KEY="check-run-$(node -e 'process.stdout.write(crypto.randomUUID())')" jq -n --arg arm "$ARM_ID" --arg verifier "$VERIFIER_ID" '{ mode: "live", verifier_id: $verifier, task: { objective: "Return only JSON, with no markdown: {\"extracted_fields\":{\"price\":number},\"summary\":string}. Extract the price from the listing and write a brief summary.", inputs_ref: { kind: "inline_json", json: { listing: "Used desk lamp, good condition. Price: 19.99." } } }, policy: { data_classes: ["public"], budget: { max_cost_usd: 1, max_runtime_s: 60 }, on_eval: [] }, routing: { required_arm_id: $arm } }' > checked-run-request.json if ! curl --fail-with-body \ --request POST \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --header "Idempotency-Key: $RUN_REQUEST_KEY" \ --header "Content-Type: application/json" \ --data @checked-run-request.json \ --output checked-execution.json \ "$MILLWORK_API_URL/executions"; then echo "The run request failed. Read checked-execution.json before retrying." >&2 exit 1 fi EXECUTION_ID="$(jq -er '.execution_id | strings | select(length > 0)' checked-execution.json)" || exit 1 ``` **Expected result:** `202` with a nonempty run ID. If the response was lost, resend the saved body with the same `RUN_REQUEST_KEY`. Changing that key starts new work. If the request was refused, follow its error code in [Errors and retries](/guides/errors-and-retries) before submitting again. Wait for a final status, then fetch the receipt: ```bash theme={null} STATUS="unknown" DEADLINE=$((SECONDS + 90)) while [ "$SECONDS" -lt "$DEADLINE" ]; do if ! curl --fail-with-body \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --output checked-status.json \ "$MILLWORK_API_URL/executions/$EXECUTION_ID"; then echo "Could not read the run status. Keep the run ID and retry this read." >&2 exit 1 fi STATUS="$(jq -er '.status | strings' checked-status.json)" || exit 1 case "$STATUS" in completed|failed|cancelled|expired) break ;; *) sleep 2 ;; esac done case "$STATUS" in completed|failed|cancelled|expired) ;; *) echo "The run is still pending. Keep the run ID and check again later." >&2; exit 1 ;; esac if ! curl --fail-with-body \ --header "Authorization: Bearer $MILLWORK_API_KEY" \ --output checked-receipt.json \ "$MILLWORK_API_URL/receipts/$EXECUTION_ID"; then echo "Could not read the receipt. Keep the run ID and retry this read." >&2 exit 1 fi ``` Confirm the receipt belongs to this run and contains a valid answer from your saved check. This test rejects missing IDs and missing verdicts; a genuine `is_correct: false` is still an answer from your check: ```bash theme={null} jq -e --arg execution "$EXECUTION_ID" --arg verifier "$VERIFIER_ID" ' ($execution | length) > 0 and ($verifier | length) > 0 and .execution_id == $execution and (.slices | type) == "array" and (.slices | length) > 0 and all(.slices[]; .verifier.verifier_id == $verifier and (.verifier.is_correct | type) == "boolean" and (.verifier.quality_score | type) == "number" and .verifier.quality_score >= 0 and .verifier.quality_score <= 1) ' checked-receipt.json >/dev/null || { echo "The receipt does not confirm a valid answer from this check. Read the run events before trying again." >&2 exit 1 } jq '.slices[] | .verifier | {verifier_id, is_correct, quality_score}' checked-receipt.json ``` **Expected result:** your saved check ID, a boolean verdict, and a numeric quality score. A negative verdict means the output did not meet your rules. An absent verdict can mean the check failed to respond; inspect `GET /v1/executions/{executionId}/events` and any `verification_checks` on the receipt. The receipt never includes the model text or `named_metrics`. For another task, save a new body and generate a new `RUN_REQUEST_KEY`. Keep the old key only for recovering the original submission. Read the run's output and understand the evidence on its receipt. Recover a failed request without starting work twice.