Connect an output check

Goal: Connect an output check you already run so Millwork can apply it to a model’s response. The API calls an output check a verifier. You are done when: your handler accepts and rejects the expected examples, and a live run’s receipt names your saved check and records its result. Your check returns two signals: whether the output meets your correctness rules (is_correct) and a separate quality score (quality_score). Millwork records these signals and uses them with the follow-up actions you allow. You own the rules; Millwork does not independently verify that they are right for your product. To use a third-party evaluator, check that its endpoint accepts and returns the formats below. You may need a handler that translates between them.

Before you begin

Your organization is approved for the private preview.
You have Node.js 20 or later, Bash or Zsh, curl, and jq.
You can add a public HTTPS route to an app you run.
You will use public information or examples created for testing.

Configure your organization API key using the private terminal instructions, or load a saved key. Those commands set SOLVERAPI_API_KEY in your terminal. Signing in through the CLI’s browser window alone does not set that variable for curl. Never paste a key into a support message or agent conversation. Use the same terminal for the rest of this guide:

: "${SOLVERAPI_API_KEY:?Load your Millwork key in this terminal first.}"
export MILLWORK_API_KEY="$SOLVERAPI_API_KEY"
export MILLWORK_API_URL="https://api.getmillwork.dev/v1"

The example handler has no authentication and makes no calls to a model or other paid service. Use it only with public or test data. To protect an endpoint, ask support to help configure the credential Millwork will send. Do not send the credential in that message. Keep any evaluator-provider key in your app; it is not the credential Millwork uses to call your handler. The final step submits a live model run and needs credit and quota. You can build, register and test the handler before choosing to run that paid step.

What Millwork sends

Every call is one HTTPS POST with a candidate field. For the model run in this guide, the candidate is the model’s response as a string. Asking for JSON does not turn that string into an object:

{
  "candidate": "{\"extracted_fields\":{\"price\":19.99},\"summary\":\"A used desk lamp in good condition.\"}"
}

The handler below parses this string before checking its fields. Invalid JSON or an unexpected shape fails the check. Other saved options, such as an agent endpoint, can return a different candidate shape; adapt the handler to the output you actually send. The request includes no run ID or check ID. Keep trusted references and acceptance rules in your handler’s configuration. Treat candidate text as data, never as code to execute. Registration and POST /v1/verifiers/{verifierId}/test send this reserved candidate:

{
  "candidate": {
    "solverapi_probe": "registration_preflight"
  }
}

What your handler must return

Return 200 with JSON. is_correct and quality_score are required on every success response, including the probe.

{
  "is_correct": true,
  "quality_score": 0.62,
  "anchor_results": {
    "has_required_field_price": true,
    "price_is_positive_number": true
  }
}

Field	Rule
`is_correct`	Boolean. Your deterministic correctness rules. No language-model judge in this path.
`quality_score`	Number in `0` to `1`. Your quality score. It never overrides `is_correct`.
`anchor_results`	Optional object of booleans. One named assertion per key. Do not put customer values in the key names; those names can reach the receipt.
`named_metrics`	Optional object of numbers. Millwork may accept it on the response. It does not appear on receipts.

A quoted "true", a quality_score outside 0 to 1, or a non-boolean anchor is a contract failure, not a failed check.

Transport facts

These are constraints of the call, not defaults you can override.

Fact	What your handler must do
Probe budget is 3 seconds	Answer the reserved probe without calling a slow backend.
Runtime budget is 10 seconds	Finish a live check in 10 seconds. Millwork does not retry and does not extend the budget.
Request is candidate-only	Do not wait for a correlation id that will not arrive.
POST may be chunked	Accept a chunked body. Millwork does not send `Content-Length`. Some serverless gateways reject that.
`named_metrics` stay off receipts	Put durable evidence in `is_correct`, `quality_score`, and `anchor_results`.
Contract validation does not gate `ready`	A reachable probe can still fail the result schema. Read `preflight.contract` (or `probe.contract` on test).

The URL must be https on a public address. Do not put a secret in the URL. Millwork does not follow redirects.

Hard rules and quality rules

Keep the two signals separate.

Hard (is_correct): schema checks, exact matches, checksums, required fields, ordered steps, other tenant-owned deterministic assertions.
Quality (quality_score): return a number from 0 to 1 on every successful response. A separate model judge is optional; the score is not. Calculate it with your scoring function, such as the example below.
A high quality score never turns a failed assertion into is_correct: true.
A low quality score never turns a passed assertion into is_correct: false.
Changing only the judge or quality function must not flip is_correct.

Test the verdict and score separately

Try these cases through the same handler you will register:

A candidate with a missing or incorrect price and a long summary must have is_correct: false, even if its quality score is high.
A candidate with the expected price and a short summary can have is_correct: true and a low quality score.
Change only the quality function. The correctness answer must stay the same.

These tests check how your handler combines the signals. They do not prove that its rules cover every mistake your product needs to catch.

Reserved probe handling

Treat only the exact reserved object as a probe:

function isReservedProbe(candidate) {
  return (
    candidate !== null
    && typeof candidate === "object"
    && !Array.isArray(candidate)
    && Object.keys(candidate).length === 1
    && candidate.solverapi_probe === "registration_preflight"
  );
}

If it is the reserved probe, return a valid negative result quickly: is_correct: false, quality_score: 0. Do not call your real check. A live candidate equal to the marker must not pass a production gate. Registration and test still accept this body; ready does not require is_correct: true.
If solverapi_probe appears in any other shape, fail closed. Do not take the probe shortcut. Never return a production pass because the marker was present. Returning is_correct: false with quality_score: 0 is the closed path.
The candidate-only request cannot prove who called you. A marker must never grant a live pass.

Build the handler

Save this example as output-check.mjs and run it with node output-check.mjs. It checks that a listing’s price matches the trusted price 19.99 configured in the handler. Its quality score measures summary length, capped at 200 characters; it does not judge the summary’s accuracy. Replace these example rules with the checks and scoring function you use for your own task.

import { createServer } from "node:http";

function isReservedProbe(candidate) {
  return (
    candidate !== null
    && typeof candidate === "object"
    && !Array.isArray(candidate)
    && Object.keys(candidate).length === 1
    && candidate.solverapi_probe === "registration_preflight"
  );
}

function containsProbeMarker(value) {
  if (value === null || typeof value !== "object") return false;
  if (Object.prototype.hasOwnProperty.call(value, "solverapi_probe")) return true;
  return Object.values(value).some(containsProbeMarker);
}

function runHardCheck(candidate) {
  const price = candidate?.extracted_fields?.price;
  const anchors = {
    has_required_field_price: typeof price === "number",
    price_is_positive_number: typeof price === "number" && price > 0,
    price_matches_listing: price === 19.99,
  };
  return { is_correct: Object.values(anchors).every(Boolean), anchor_results: anchors };
}

function scoreQuality(candidate) {
  const text = typeof candidate.summary === "string" ? candidate.summary : "";
  return Math.min(1, text.length / 200);
}

async function readJson(request) {
  const chunks = [];
  for await (const chunk of request) chunks.push(chunk);
  return JSON.parse(Buffer.concat(chunks).toString("utf8"));
}

const server = createServer(async (request, response) => {
  const send = (status, body) => {
    response.writeHead(status, { "content-type": "application/json" });
    response.end(JSON.stringify(body));
  };

  if (request.method !== "POST") {
    send(405, { is_correct: false, quality_score: 0 });
    return;
  }

  let payload;
  try {
    payload = await readJson(request);
  } catch {
    send(400, { is_correct: false, quality_score: 0 });
    return;
  }

  let candidate = payload?.candidate;
  if (isReservedProbe(candidate)) {
    send(200, { is_correct: false, quality_score: 0 });
    return;
  }
  if (containsProbeMarker(candidate)) {
    send(200, { is_correct: false, quality_score: 0 });
    return;
  }

  // Model output is text, even when it contains JSON.
  if (typeof candidate === "string") {
    try {
      candidate = JSON.parse(candidate);
    } catch {
      send(200, { is_correct: false, quality_score: 0 });
      return;
    }
  }
  if (candidate === null || typeof candidate !== "object"
      || Array.isArray(candidate) || containsProbeMarker(candidate)) {
    send(200, { is_correct: false, quality_score: 0 });
    return;
  }

  const hard = runHardCheck(candidate);
  send(200, {
    is_correct: hard.is_correct,
    quality_score: scoreQuality(candidate),
    anchor_results: hard.anchor_results,
  });
});

server.listen(8080);

Serve this route over public HTTPS. Terminate TLS in front of this process if your app already does. Register the public https URL, not http://127.0.0.1.

Deploy and register the handler

Deploy the handler in your app and confirm that its public HTTPS URL is reachable. Millwork does not deploy it for you. Keep the route available; when you change its scoring logic, update the registered version and test it again. If its authentication changes, arrange the new credential binding with support before using it for another run. Replace this example URL with your handler’s URL. A request key is the value sent in the Idempotency-Key header so retrying the same request does not repeat the work. Generate one for registration and keep it with the body:

export VERIFIER_URL="https://your-app.example/output-check"
export REGISTER_REQUEST_KEY="check-register-$(node -e 'process.stdout.write(crypto.randomUUID())')"

jq -n --arg url "$VERIFIER_URL" '{
  display_name: "Listing extraction check",
  version: "1.0",
  kind: "endpoint",
  endpoint: { url: $url, auth_ref: "" },
  input_data_classes: ["public"],
  scoring: { correctness: "boolean_anchors", quality: "scalar_0_1" }
}' > verifier-request.json

if ! curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer $MILLWORK_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: $REGISTER_REQUEST_KEY" \
  --data @verifier-request.json \
  --output verifier-response.json \
  "$MILLWORK_API_URL/verifiers"; then
  echo "Registration failed. Read verifier-response.json before continuing." >&2
  exit 1
fi

VERIFIER_ID="$(jq -er '.verifier_id | strings | select(length > 0)' verifier-response.json)" || exit 1
export VERIFIER_ID

Expected result: 201 with a nonempty verifier_id, hash, status, and preflight. auth_ref: "" means this endpoint needs no authentication. Keep VERIFIER_ID; the live request below uses it. If the response was lost, resend the same body with the same REGISTER_REQUEST_KEY. Reusing that key returns the stored response, including a stored error; after correcting a rejected request, use a new key. Test the saved check with a fresh request key. Reuse this key only to recover this particular test request, not to make a new observation after fixing the handler:

export TEST_REQUEST_KEY="check-test-$(node -e 'process.stdout.write(crypto.randomUUID())')"
if ! curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer $MILLWORK_API_KEY" \
  --header "Idempotency-Key: $TEST_REQUEST_KEY" \
  --output check-test.json \
  "$MILLWORK_API_URL/verifiers/$VERIFIER_ID/test"; then
  echo "The test request failed. Read check-test.json before continuing." >&2
  exit 1
fi
jq -e '.status == "ready" and .probe.outcome == "reachable"
  and .probe.contract.validated == true' check-test.json >/dev/null || exit 1

Expected result: ready, reachable, and a validated response. The example’s reserved probe returns is_correct: false and quality_score: 0; that negative answer is intentional. ready means the probe had no reachability or authentication failure. It does not certify your correctness rules, and it does not by itself mean the response matched the required schema.

What the recorded hash covers

The hash covers the registered definition: display name, version, URL, input data classes, and scoring shape. It does not fingerprint deployed code, models, prompts, datasets, or dependencies behind a stable URL. Changing scoring logic without bumping version (or another hashed field) does not necessarily produce a new hash on later receipts. Versioning is your obligation. Repeated probes can falsify a determinism claim. They cannot prove the absence of a language model, a cache, or mutable external state. A queue-existence or URL-resolvability check depends on changing external state. Determinism, correctness coverage, independence, and availability are four different properties. Observed repeatability is not independent verification.

Describe your check’s correctness rules

The API accepts an optional correctness_declaration with { "method": "deterministic" } when you register or update a check. This records your organization’s statement; Millwork does not independently verify it. A declaration-only update changes the revision, not the definition hash. New evaluations record the declaration in effect at that time. A receipt can show “Tenant declared deterministic correctness; not independently verified.” Older receipts do not gain a declaration when you add one later. See the API reference for registration, updates, and reads.

If registration or test fails

What you see	What it means	What to do
`status` `degraded`, reason `dns_resolution_failed` or unreachable	Millwork could not reach the public HTTPS URL	Confirm the hostname, TLS, and that the address is public. Retest.
HTTP `401` or `403` on a credentialed probe	Authentication failed	Correct the adapter credential your handler expects. Retest.
HTTP `2xx` and `contract.validated` false	Reachable, wrong body	Fix `is_correct`, `quality_score`, or `anchor_results`. Retest.
Probe exceeds 3 seconds	The reserved shortcut did not run, or a backend ran during probe	Short-circuit the exact reserved candidate. Retest.
The output check takes longer than 10 seconds	The check timed out	Make the handler respond within 10 seconds, then test it again. Millwork does not retry the check.

After each correction, call POST /v1/verifiers/{verifierId}/test again. Do not assume a later run will retest for you.

Use the check on a live run

A live run charges a platform fee when accepted and may incur separate model usage. It needs credit and quota. Review model access and billing before continuing. The handler in this example adds no model call of its own. You need a saved model (the API calls it an arm). If you do not have one, open Run a model and complete Choose a model from your catalog and Save the model to your organization. Return here before its Submit the run step. Keep its ARM_ID in this terminal. You do not need to run its example task. Check the saved model and both IDs before preparing the paid request:

: "${VERIFIER_ID:?Register the output check first.}"
: "${ARM_ID:?Choose and save a model, then return here with ARM_ID.}"
if ! curl --fail-with-body \
  --header "Authorization: Bearer $MILLWORK_API_KEY" \
  --output saved-model.json \
  "$MILLWORK_API_URL/arms/$ARM_ID"; then
  echo "Could not read the saved model. Read saved-model.json." >&2
  exit 1
fi
jq -e --arg arm "$ARM_ID" '.arm_id == $arm and .status == "ready"' saved-model.json >/dev/null || exit 1

The task below asks for the JSON text the handler expects. It explicitly attaches your VERIFIER_ID; omitting that field would use a basic output-presence check instead. on_eval: [] requests no repair or fallback action after the check. The model-usage budget stops further calls after recorded usage reaches it; a call already started can finish above it. The platform fee is separate.

export RUN_REQUEST_KEY="check-run-$(node -e 'process.stdout.write(crypto.randomUUID())')"
jq -n --arg arm "$ARM_ID" --arg verifier "$VERIFIER_ID" '{
  mode: "live",
  verifier_id: $verifier,
  task: {
    objective: "Return only JSON, with no markdown: {\"extracted_fields\":{\"price\":number},\"summary\":string}. Extract the price from the listing and write a brief summary.",
    inputs_ref: { kind: "inline_json", json: { listing: "Used desk lamp, good condition. Price: 19.99." } }
  },
  policy: { data_classes: ["public"], budget: { max_cost_usd: 1, max_runtime_s: 60 }, on_eval: [] },
  routing: { required_arm_id: $arm }
}' > checked-run-request.json

if ! curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer $MILLWORK_API_KEY" \
  --header "Idempotency-Key: $RUN_REQUEST_KEY" \
  --header "Content-Type: application/json" \
  --data @checked-run-request.json \
  --output checked-execution.json \
  "$MILLWORK_API_URL/executions"; then
  echo "The run request failed. Read checked-execution.json before retrying." >&2
  exit 1
fi
EXECUTION_ID="$(jq -er '.execution_id | strings | select(length > 0)' checked-execution.json)" || exit 1

Expected result: 202 with a nonempty run ID. If the response was lost, resend the saved body with the same RUN_REQUEST_KEY. Changing that key starts new work. If the request was refused, follow its error code in Errors and retries before submitting again. Wait for a final status, then fetch the receipt:

STATUS="unknown"
DEADLINE=$((SECONDS + 90))
while [ "$SECONDS" -lt "$DEADLINE" ]; do
  if ! curl --fail-with-body \
    --header "Authorization: Bearer $MILLWORK_API_KEY" \
    --output checked-status.json \
    "$MILLWORK_API_URL/executions/$EXECUTION_ID"; then
    echo "Could not read the run status. Keep the run ID and retry this read." >&2
    exit 1
  fi
  STATUS="$(jq -er '.status | strings' checked-status.json)" || exit 1
  case "$STATUS" in
    completed|failed|cancelled|expired) break ;;
    *) sleep 2 ;;
  esac
done
case "$STATUS" in
  completed|failed|cancelled|expired) ;;
  *) echo "The run is still pending. Keep the run ID and check again later." >&2; exit 1 ;;
esac
if ! curl --fail-with-body \
  --header "Authorization: Bearer $MILLWORK_API_KEY" \
  --output checked-receipt.json \
  "$MILLWORK_API_URL/receipts/$EXECUTION_ID"; then
  echo "Could not read the receipt. Keep the run ID and retry this read." >&2
  exit 1
fi

Confirm the receipt belongs to this run and contains a valid answer from your saved check. This test rejects missing IDs and missing verdicts; a genuine is_correct: false is still an answer from your check:

jq -e --arg execution "$EXECUTION_ID" --arg verifier "$VERIFIER_ID" '
  ($execution | length) > 0 and ($verifier | length) > 0
  and .execution_id == $execution
  and (.slices | type) == "array" and (.slices | length) > 0
  and all(.slices[];
    .verifier.verifier_id == $verifier
    and (.verifier.is_correct | type) == "boolean"
    and (.verifier.quality_score | type) == "number"
    and .verifier.quality_score >= 0 and .verifier.quality_score <= 1)
' checked-receipt.json >/dev/null || {
  echo "The receipt does not confirm a valid answer from this check. Read the run events before trying again." >&2
  exit 1
}
jq '.slices[] | .verifier | {verifier_id, is_correct, quality_score}' checked-receipt.json

Expected result: your saved check ID, a boolean verdict, and a numeric quality score. A negative verdict means the output did not meet your rules. An absent verdict can mean the check failed to respond; inspect GET /v1/executions/{executionId}/events and any verification_checks on the receipt. The receipt never includes the model text or named_metrics. For another task, save a new body and generate a new RUN_REQUEST_KEY. Keep the old key only for recovering the original submission.

Results and receipts

Read the run’s output and understand the evidence on its receipt.

Errors and retries

Recover a failed request without starting work twice.

Get started

Build

Build safely

SDK and MCP

Connect an output check

Before you begin

What Millwork sends

What your handler must return

Transport facts

Hard rules and quality rules

Test the verdict and score separately

Reserved probe handling

Build the handler

Deploy and register the handler

What the recorded hash covers

Describe your check’s correctness rules

If registration or test fails

Use the check on a live run

Results and receipts

Errors and retries

​Before you begin

​What Millwork sends

​What your handler must return

​Transport facts

​Hard rules and quality rules

​Test the verdict and score separately

​Reserved probe handling

​Build the handler

​Deploy and register the handler

​What the recorded hash covers

​Describe your check’s correctness rules

​If registration or test fails

​Use the check on a live run

Results and receipts

Errors and retries

Before you begin

What Millwork sends

What your handler must return

Transport facts

Hard rules and quality rules

Test the verdict and score separately

Reserved probe handling

Build the handler

Deploy and register the handler

What the recorded hash covers

Describe your check’s correctness rules

If registration or test fails

Use the check on a live run