Run a DSPy program with a Jev check

Goal: run your DSPy program and Jev check through Millwork when you need an approved run and a receipt.

See it in a warranty answer

In the included offline example, someone asks whether a refurbished laptop has a warranty. DSPy first proposes a lifetime warranty, but the trusted policy promises 12 months. A scripted Jev decision scores the first answer low, so DSPy tries again. Your output check asks whether the cited policy passage supports the final answer. This diagram shows the checks you try offline and the receipt produced when you run the same endpoints through Millwork. For an approved live run, Millwork calls the same Python app and records which saved agent and output check handled it, plus the verdict. Your DSPy program and acceptance rules run in your Python app. The model provider and TypeSafe receive the inputs your app sends them; Millwork’s receipt stores neither the question, answer, nor passage. The saved agent and check can handle later approved runs; your app chooses which questions to route.

Where else this fits

The same pattern works for short answers grounded in source passages your team controls. The warranty example is included; each idea below needs its own source and offline cases:

Customer support

Bring the current return policy. Test a correct answer, a wrong return window, and a missing quote.

Research summaries

Bring approved source excerpts. Check one cited claim at a time; test a misquote and a contradiction.

Developer docs

Bring the reference for the right API version. Test a supported claim and a quote from an older release.

For each adaptation, replace the sample answerer and passage store with your DSPy module and trusted sources. Your app controls source freshness and access. The check can judge whether a cited passage supports an answer; it cannot prove the answer completes the user’s whole task. For other kinds of Jev decisions, explore TypeSafe’s use-case map; those need their own question, evidence, and tests.

Start with the offline example

No account or key. Run the included cases before adapting the program or sending live traffic.

You are done with the offline example when: offline_cases.py reports 4 passed, 0 failed and check_locally.py reports 14 passed, 0 failed. After an approved live run, its receipt names your saved agent, output check, and verdict; the usage report separates your answers and Jev decisions from Millwork’s accepted runs. TypeSafe’s Jev returns a probability that a passage supports an answer. DSPy’s BestOfN tries up to your set limit and stops when an answer reaches your threshold. Your output check applies exact quote rules and a separate Jev decision to the final answer.

Before you begin

For the offline example: Python 3.10 or newer. It needs no account, key or credit. For the endpoint test and the live run:

Node.js and the Millwork CLI release that Set up with the CLI installs, plus curl and jq. Run millwork --version --json first and confirm it reports that release; use it for every CLI command on this page.
An organization API key in your secret store, available as SOLVERAPI_API_KEY, as described in Get your first model answer.
A TypeSafe account for Jev, and an account with the model provider your DSPy program already uses.
Millwork credit and quota for one run, and an administrator who can register your agent endpoint’s key and give you a credential handle, a reference to that key. Without that handle you can complete only the offline steps.

Give this page to your coding agent. Ask it to run the offline example, return both test results and name the files it would adapt, then stop at each Approval line. The named owner approves that action and enters keys privately.

Run the example offline

The example is a small policy-answering app. program.py holds a DSPy module standing in for yours, wrapped in a bounded BestOfN. check.py holds the output check, served through a Python output-check adapter. Copy the downloader into dspy-example.py, review it, then run it. It creates a dspy-jev-governed-run directory and checks every file against the manifest before writing anything.

Required: save as dspy-example.py

# Generated from the DSPy + Jev example. Review before running.
import hashlib
import json
import pathlib
import re
import urllib.request

target = pathlib.Path("dspy-jev-governed-run").resolve()
url = "https://docs.getmillwork.dev/downloads/dspy-jev-governed-run.json"
expected = "86e8ba2fd0c548c255b99b8a8c7b114e6b3e2d0378caa2263fe9f4cd29c1b949"
if target.exists():
    raise SystemExit("Target directory already exists")
with urllib.request.urlopen(url, timeout=30) as response:
    data = response.read(300001)
if len(data) > 300000:
    raise SystemExit("Example bundle is too large")
bundle = json.loads(data.decode("utf-8"))
files = bundle.get("files")
if not isinstance(files, list) or len(files) != 25:
    raise SystemExit("Unexpected file inventory")
seen, checked = set(), []
for entry in files:
    name, digest, content = entry.get("path"), entry.get("sha256"), entry.get("content")
    parts = name.split("/") if isinstance(name, str) else []
    if not parts or name.startswith("/") or "\\" in name or any(part in ("", ".", "..") for part in parts) or name in seen:
        raise SystemExit("Unsafe file path")
    if not isinstance(digest, str) or not re.fullmatch(r"[0-9a-f]{64}", digest) or not isinstance(content, str):
        raise SystemExit("Invalid file entry")
    path = (target / name).resolve()
    if target not in path.parents:
        raise SystemExit("Unsafe target path")
    raw = content.encode("utf-8")
    if hashlib.sha256(raw).hexdigest() != digest:
        raise SystemExit(f"Checksum mismatch: {name}")
    seen.add(name)
    checked.append((name, digest, path, raw))
manifest = "".join(f"{name}\0{digest}\n" for name, digest, _, _ in checked)
if hashlib.sha256(manifest.encode("utf-8")).hexdigest() != expected or bundle.get("aggregate_sha256") != expected:
    raise SystemExit("Bundle manifest mismatch")
for _name, _digest, path, raw in checked:
    path.parent.mkdir(parents=True, exist_ok=True)
    with open(path, "xb") as handle:
        handle.write(raw)
print(f"Created {target} with {len(checked)} checked files.")

python3 dspy-example.py
cd dspy-jev-governed-run
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python offline_cases.py
python check_locally.py --check offline_check.py

Expected: 4 passed, 0 failed and 14 passed, 0 failed. offline_cases.py walks four questions through both endpoints: an answer supported on the first attempt, one supported on the second, an abstention after three unsupported attempts, and a Jev outage. check_locally.py checks supported, contradicted, missing-source, low-confidence, misquoted and abstained answers, and an unavailable check. It sends each answer again as model text, which must get exactly the same result, and confirms that the quality score never changes a verdict. Scripted answers stand in for your model and for Jev, so neither command needs a key or makes a paid call. Both still run DSPy’s Predict, BestOfN and the code that turns Jev’s answer into a decision, as production does. Run every later command from this dspy-jev-governed-run directory unless a step says otherwise.

See one question, answer and verdict

The same app serves both endpoints on your machine with the scripted answers. The two keys below are local demo values that only this offline server accepts; never use them for a deployed endpoint.

export MILLWORK_AGENT_KEYS="offline-demo-agent" MILLWORK_VERIFIER_KEYS="offline-demo-check"
python app.py --offline &
APP_PID=$!
READY=""
for attempt in $(seq 60); do
  if curl -sf http://127.0.0.1:8080/healthz > /dev/null; then READY=yes; break; fi
  kill -0 "$APP_PID" 2> /dev/null || break
  sleep 0.5
done
if [ -n "$READY" ]; then
  curl -s --header "Authorization: Bearer offline-demo-agent" \
    --header "Content-Type: application/json" \
    --data '{"task":{"objective":"Can I return an unused item three weeks after delivery?"}}' \
    http://127.0.0.1:8080/millwork/agent | tee answer.json
  jq '{candidate}' answer.json | curl -s --header "Authorization: Bearer offline-demo-check" \
    --header "Content-Type: application/json" --data @- \
    http://127.0.0.1:8080/millwork/check
else
  echo "app.py did not answer /healthz within 30 seconds; see its output above."
fi
kill "$APP_PID" 2> /dev/null

Expected: the agent returns an answer with its source_id and quote, and the check returns "is_correct": true with four named results. The app also prints one count record per endpoint; those lines hold numbers and outcome codes only.

Put your program and evidence in place

Change these files and keep the rest:

What you already have	Where it goes
Your DSPy module	Replace `PolicyAnswerer` in `program.py`, or pass a module you already built or loaded to `build_live_components(..., answerer=your_module)`. It takes `question` and `passages`, returns `answer`, `source_id` and `quote`, and makes its model calls through DSPy predictors. If it makes more than one call per answer, also pass `model_calls_per_attempt`.
The passages your answers rely on	Replace the file-backed store in `trusted_store.py` with the source you already trust. The check reads each cited passage from here, never from the answer.
Your Jev question	Edit `SourceSupport` in `jev.py`. It returns a probability, using TypeSafe’s `Noul` question type.
Your acceptance rule	`support.py`: the exact checks, then `SUPPORT_THRESHOLD` (0.8).
How many answers to try	`attempts` in `GovernedPolicyProgram`: 3 by default, 5 at most. Each attempt can produce one answer and at most one Jev decision. The run’s model calls are capped at `attempts` times `model_calls_per_attempt` (1 by default), and all attempts must finish inside the program’s 25-second deadline.

If the quote is not in the cited passage, the check rejects the answer without asking Jev. It passes an answer only when the exact checks hold and Jev’s probability reaches your threshold.Add your own offline cases before going live. The scripted model answers live in offline.py: give each question one scripted answer per attempt, and include every output field your module declares, such as reasoning for dspy.ChainOfThought. Add a JourneyCase in offline_cases.py for each new question, and your check’s cases to check_cases.json and heldout_cases.json. Include a pass, a contradiction, a missing source, a low-confidence answer and an unavailable check. Then rerun both offline commands. Offline runs use scripted answers, so they test your check and your wiring, not your prompt; judge the prompt on the live path.

Choose your threshold with Jev

Cost: from this step on, some commands cost money. TypeSafe bills each Jev decision, and your model provider bills each answer your program generates, including every BestOfN attempt. Millwork charges its platform fee only when it accepts a live run, which happens in the last steps. Model access and billing gives the current amount and the full rule. Local cases, endpoint tests, BestOfN attempts and Jev calls on their own are not Millwork runs and carry no platform fee.The threshold is a decision rule you choose from labelled answers, not a calibrated chance of being right. evaluate_quality.py compares the check with the exact-quote check alone. It shows false acceptance and false rejection at every threshold from 0.50 to 0.95, with the latency and TypeSafe requests for the whole run. First see the report’s shape with scripted answers:

python evaluate_quality.py --offline

Its numbers are not a measurement: the scripted answers give every labelled answer the same probability, 0.3, so every row rejects all supported answers. For the real measurement, run python evaluate_quality.py --live. It prints how many TypeSafe requests it needs and stops.Approval: the TypeSafe account owner approves that many requests. Then the same person loads the key in their own terminal, so it never appears in a command, a file or a chat:

read -rs TYPESAFE_API_KEY && export TYPESAFE_API_KEY
APPROVED_COUNT="<the count the TypeSafe account owner approved>"
python evaluate_quality.py --live --authorize-typesafe-requests "$APPROVED_COUNT"

Expected: a report marked "evidence": "live". Set SUPPORT_THRESHOLD in support.py to the jev_check.by_threshold row whose false acceptance and false rejection your organization accepts. If Jev times out, refuses or returns a malformed answer, that case counts as a technical failure with no probability. The report holds case IDs and numbers, never answer or passage text.

Deploy both endpoints over HTTPS

Deployment can incur hosting charges. The endpoint test at the end of this step sends paid Jev requests.Deploy through the application or host your team already owns, following the deployment step of Build and connect an output check for HTTPS, access and test approvals. Copy the dspy-jev-governed-run directory to the host and install requirements.txt there. Configure these values in the host’s secret settings:

Setting	Holds
`MILLWORK_AGENT_KEYS`	The key Millwork presents to `/millwork/agent`.
`MILLWORK_VERIFIER_KEYS`	A different key, for `/millwork/check`.
`GENERATOR_MODEL`	The language model (LM) your DSPy program uses, such as `openai/gpt-4o-mini`, plus that provider’s own key setting.
`TYPESAFE_API_KEY`	Read by the TypeSafe library; the example never reads or logs it.
`MILLWORK_COUNTER_LOG`	A file for the count records on a volume that survives restarts, such as `/var/log/app/counters.jsonl`. When it is unset, the records go to the app’s error output; export them from your log platform instead.

Neither endpoint key is a Millwork organization API key. Both are read on each request, so you can rotate them by changing the settings.Run the app with a production WSGI server behind your HTTPS ingress, reachable only from that ingress. wsgi.py builds the app once per worker:

gunicorn --worker-class gthread --threads 8 --timeout 60 --bind 0.0.0.0:8080 wsgi:application

Use a threaded worker: each agent call holds a thread for up to the 25-second program deadline. python app.py starts a single-process server for the offline walkthrough only. If your web app is ASGI, such as FastAPI, run this as its own service and route /millwork/agent and /millwork/check to it from your ingress. GET /healthz reports the example, adapter, DSPy and Python versions only.Then test the check endpoint with the endpoint test kit: Node scripts, created by the CLI, that send labelled requests to your check, including cases meant to fail. check-cases.mjs gives the kit each case in check_cases.json once, as model text, and checks its verdict and named results. The exact match between the JSON and model-text forms of a case was already checked offline by check_locally.py; against a live endpoint each form would be a separate Jev request, and two requests need not return the same probability.Only cases whose quote is in their passage reach Jev. Count the requests one kit run sends with your current cases:

python kit_request_bound.py

Approval: the endpoint owner and the TypeSafe account owner approve the printed count for each kit run you plan. Ask again before any run beyond that. Then create the kit inside dspy-jev-governed-run, and have the endpoint owner load the check key with read -rs MILLWORK_KIT_ENDPOINT_KEY && export MILLWORK_KIT_ENDPOINT_KEY:

millwork verifier init output-check --json
cd output-check
node check-endpoint.mjs \
  --deployed https://app.example.com/millwork/check \
  --authorize-endpoint-test \
  --check ../check-cases.mjs \
  --access authenticated \
  --json
cd ..

Expected: passed is true and summary.failed is 0; the local-only evaluator-unavailable case shows as not applicable. Passing means the deployed endpoint answers Millwork’s contract with your access settings, and live Jev gives each labelled case the verdict you expect. If a case gets a different verdict, your labelled case and live Jev disagree: fix the case or your threshold offline, then get approval before testing again. The kit is a test tool on your workstation; the deployed app does not need Node.

Save your agent

The API calls a saved agent an arm. Ask your administrator to register the agent endpoint’s key (MILLWORK_AGENT_KEYS) and give you its credential handle. The key itself never appears in your request. Save this body as dspy-agent.json:

{
  "kind": "agent",
  "display_name": "DSPy policy answerer",
  "endpoint": {
    "url": "https://app.example.com/millwork/agent",
    "auth_ref": "<registered-agent-credential-handle>"
  },
  "capability_tags": ["policy_answers"],
  "data_class_grants": ["sandbox"],
  "lifecycle": { "long_running": false, "cancellable": false, "max_runtime_s": 30 }
}

Approval: saving the agent sends one authenticated HEAD request, a probe, to the agent endpoint, which answers it without running your program. The endpoint owner approves that probe. Then save the agent with your organization API key:

set -o pipefail
curl --fail-with-body --request POST \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --header "Content-Type: application/json" \
  --data @dspy-agent.json \
  --output dspy-agent-response.json \
  "https://api.getmillwork.dev/v1/arms" &&
  jq -e '{arm_id, status, status_reason}' dspy-agent-response.json

Expected: status: "ready" and an arm_id, which the run request reads from dspy-agent-response.json. A degraded status with authentication_failed means the handle’s key and MILLWORK_AGENT_KEYS differ.

Connect your output check

The API calls an output check a verifier (verifier_id). Connecting and testing the output check have no Millwork platform fee, and neither calls Jev. Connecting sends two test requests to your endpoint: a probe without the key, which the adapter refuses, then a test that checks the key. millwork verifier test sends one that checks the endpoint contract. CUSTOMER_APP_ORIGIN is the Millwork app you sign in to, not your endpoint.Approval: the endpoint owner approves these three test requests. Then:

export VERIFIER_URL="https://app.example.com/millwork/check"
export CUSTOMER_APP_ORIGIN="https://app.getmillwork.dev"
millwork verifier connect \
  --endpoint "$VERIFIER_URL" \
  --access managed \
  --name "Policy answer check" \
  --version "1.0.0" \
  --stop-days 90 \
  --connect-only

The endpoint owner enters the MILLWORK_VERIFIER_KEYS key in the hidden terminal prompt; the command then prints Verifier <id> connected. Without a private terminal, it returns state: "action_required" with a verifier_id, a private continue_url and an intent_id. Open continue_url in your browser, signed in to the same organization, enter the key there, then finish the connection:

millwork verifier continue \
  --verifier-id "<verifier_id from action_required>" \
  --intent-id "<intent_id from action_required>" \
  --json

Keep the verifier_id once the command printed Verifier <id> connected. or the continue output shows connection.status: "active". Then test it:

export VERIFIER_ID="<the connected verifier_id>"
millwork verifier test \
  --verifier-id "$VERIFIER_ID" \
  --idempotency-key "verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')" \
  --json

Expected: headline: "ready", probe.contract.validated: true and the same verifier_id. The reserved probe may return is_correct: false on purpose; that does not make the endpoint unusable. To repair a connection, see protected-connection recovery.

Approve one run and read the receipt

Cost: Millwork’s platform fee, charged when it accepts the run; see Model access and billing for the current amount. With the default three attempts, one run makes at most three model calls and sends at most four Jev decisions: one per attempt and one in the check. The request’s max_cost_usd limits only the model spend Millwork records, and this example reports none, so it does not limit your provider or TypeSafe bills; the program’s attempts and deadline limit those.Build the request. run_ref is a new random ID the usage report uses to match the run to its counts, so it never holds customer data. The request key lets Millwork recognize a retry, so save it for any retry:

RUN_REF="run-$(node -e 'process.stdout.write(crypto.randomUUID())')"
RUN_KEY="run-key-$(node -e 'process.stdout.write(crypto.randomUUID())')"
printf '%s\n' "$RUN_KEY" > run-key.txt
jq -n \
  --arg run_ref "$RUN_REF" \
  --arg arm "$(jq -er '.arm_id' dspy-agent-response.json)" \
  --arg check "$VERIFIER_ID" \
  '{mode: "live",
    task: {objective: "Can I return an unused item three weeks after delivery?",
           inputs_ref: {kind: "inline_json", json: {run_ref: $run_ref}}},
    policy: {data_classes: ["sandbox"], budget: {max_cost_usd: 0.05, max_runtime_s: 60}, on_eval: []},
    routing: {required_arm_id: $arm},
    verifier_id: $check}' > run-request.json
cat run-request.json

routing.required_arm_id runs the task on your saved agent, and verifier_id selects your check. Without verifier_id, Millwork would apply only a basic check that output is present.Approval: a person who can authorize paid runs reviews run-request.json and approves this one run. Then submit it:

set -o pipefail
curl --fail-with-body --request POST \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --header "Idempotency-Key: $(cat run-key.txt)" \
  --header "Content-Type: application/json" \
  --data @run-request.json \
  --output run-execution.json \
  "https://api.getmillwork.dev/v1/executions" &&
  jq -e '{execution_id, status}' run-execution.json

Expected: a run ID and a queued status, not a finished run. If the response is lost, send the same command again: the saved request key returns the same run instead of starting a second one. Check the status until it is completed, failed, cancelled or expired, as Wait for a final status describes:

EXECUTION_ID="$(jq -er '.execution_id' run-execution.json)"
curl --fail-with-body \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  "https://api.getmillwork.dev/v1/executions/$EXECUTION_ID" | jq -er '.status'

Then read the receipt:

curl --fail-with-body \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  "https://api.getmillwork.dev/v1/receipts/$EXECUTION_ID" |
  jq -e '.slices[0] | {agent: .route.selected_arm, check: .verifier.verifier_id, verdict: .verifier.is_correct, results: .verifier.anchor_results, recorded_check: .recorded_check}'

Expected: agent is your arm_id, check is your verifier_id, and verdict is the check’s decision with its four named results. The receipt shows the agent attempt at $0 because the example reports no cost to Millwork; your provider and TypeSafe bills carry that spend.What other outcomes look like:

What you see	What it means
`verdict: false`, status `completed`	The check rejected the answer, or your program abstained. With `on_eval: []` Millwork does not retry. It records `needs_review` on the run’s lifecycle event and sends the `execution.needs_review` webhook if you subscribe to it.
No verdict, `recorded_check.state` is `customer_unavailable`	Your check could not decide, for example Jev did not answer in time. Its `failure_class` says why.
No verdict, `recorded_check.state` is `terminal_no_evaluation`	The run failed before the check ran. A `dispatch_timeout` reason means the agent call ran past its time limit.

Results and receipts explains the verdict, quality score and named results.

Count what the run used

Copy the count file that MILLWORK_COUNTER_LOG names from your host into this directory, or export the records from your log platform. Save Millwork’s usage report for the month the run was accepted, read from the run’s own record:

RUN_MONTH="$(jq -er '.created_at[0:7]' run-execution.json)"
curl --fail-with-body \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --output usage.json \
  "https://api.getmillwork.dev/v1/usage?period=$RUN_MONTH"
python usage_report.py \
  --counters counters.jsonl \
  --run run-request.json run-execution.json \
  --usage usage.json

Expected: an executions entry for your run ID with the counts your app recorded, and a millwork section with the runs Millwork accepted in the whole period. The report skips log lines that are not count records.

Count	Where it appears	Billed by
Model calls your program made	`generator_calls` in the run’s entry	Your model provider
Answers your program generated	`best_of_n_candidates_generated` in the run’s entry	Included in model calls
Jev decisions sent, in the program and the check	`typesafe_requests_sent` in the run’s entry	TypeSafe
Runs Millwork accepted in the period	`millwork.accepted_executions`	Millwork

generator_calls counts every model call, including ones whose output DSPy could not parse (generation_failures); a call beyond the run’s cap or after the deadline is refused, not sent. typesafe_requests_sent includes requests that failed, which typesafe_failures counts again on their own. Decisions answered from DSPy’s cache are counted in typesafe_cache_hits and are not sent. Count records hold numbers, outcome codes and your run_ref only: never a question, answer, passage or key. Records from local cases and endpoint tests have no matching run and appear under unattributed.To use this path for another eligible question, keep the saved agent and output check. Build a new request with a fresh run_ref and request key, and get approval for that paid run. Your app decides which questions to route; finishing this example does not route future questions automatically.

Time limits

Millwork waits at most 30 seconds for the agent call, or less when the saved agent’s max_runtime_s or the run’s remaining time is smaller. The example’s program deadline, 25 seconds, usually ends first and answers 504 program_timeout, a technical failure with no verdict. A run that waited long in the queue can reach Millwork’s limit first instead. A program that needs longer is not suited to this recipe. Each model answer has an 8-second limit and each Jev decision a 6-second limit, with no automatic retries. After a 504, no new model answer or Jev decision starts. A call already in flight finishes, and your app then writes a late-work record with its counts, which the usage report adds to the run. The check’s Jev decision runs inside Millwork’s 10-second wait for a check.

What the receipt does not show

The receipt records the check’s identity, verdict, quality score and named results. It does not keep the question, the answer, the passage or Jev’s response. The check sees only the answer. It checks whether the cited passage supports the answer, not whether the answer completed the customer’s task.

Recover without starting work twice

Symptom	What to do
Saving the agent returns `authentication_failed`	Register the handle again with the exact key in `MILLWORK_AGENT_KEYS`.
The receipt shows `dispatch_timeout`, or your logs show `program_timeout`	Lower the BestOfN attempts or your generator’s latency, then send a new run.
The receipt shows `customer_unavailable`	Your check could not decide. Check TypeSafe’s status and your key, then send a new run.
The agent endpoint answers `503 ranking_unavailable`	Jev did not answer while your program scored its answers. Check TypeSafe’s status and your key, then send a new run.
The check test fails	Follow endpoint-test recovery.
A submission’s outcome is unknown	Send the same command again with the request key saved in `run-key.txt`. Read Errors and retries.

For a Jev check without DSPy, start from Check citations with Jev.

Output checks

Review jobs

Operations recipes

Run a DSPy program with a Jev check

See it in a warranty answer

Where else this fits

Customer support

Research summaries

Developer docs

Start with the offline example

Before you begin

Time limits

What the receipt does not show

Recover without starting work twice

​See it in a warranty answer

​Where else this fits

Customer support

Research summaries

Developer docs

Start with the offline example

​Before you begin

​Time limits

​What the receipt does not show

​Recover without starting work twice

See it in a warranty answer

Where else this fits

Before you begin

Time limits

What the receipt does not show

Recover without starting work twice