> ## Documentation Index > Fetch the complete documentation index at: https://docs.getmillwork.dev/llms.txt > Use this file to discover all available pages before exploring further. # Check a CodeRabbit review > Check a completed CodeRabbit review of a pinned commit and record the decision on a Millwork run. **Goal:** check a completed CodeRabbit review of a specific commit in a Millwork run. **You are done with the offline example when:** its pass, reject and no-verdict cases work. After an approved live run, read the receipt for the final decision. Use this when CodeRabbit already reviews your changes and you want its result to inform a Millwork run. Your output check applies rules you choose, and the run receipt records what happened. If your pull request check already makes the same decision and you need no Millwork run, keep that workflow.
Run Recipe D, download the checked example once, then try its local cases without keys or CodeRabbit calls.
## How it works Suppose a coding worker changes an authorization rule. CodeRabbit reviews the committed change before the Millwork run. During the run, your agent endpoint returns an evidence ID; the saved review stays in your store. This diagram shows how the review becomes one output-check result. ```mermaid theme={null} flowchart TD A["CodeRabbit reviews the commit"] --> B["Your runner saves a signed review record"] B --> C["Your agent endpoint returns the evidence ID"] C --> D["Your output check applies your rules"] D --> E["Millwork records the result in a receipt"] classDef external fill:#F3F0F8,stroke:#9A86B8,color:#211F1B,stroke-width:1px classDef customer fill:#EEF4F6,stroke:#7895A0,color:#211F1B,stroke-width:1px classDef decision fill:#F8F6F3,stroke:#B92B2B,color:#211F1B,stroke-width:1px classDef receipt fill:#EDF6EF,stroke:#4E8A60,color:#211F1B,stroke-width:1px class A external class B,C customer class D decision class E receipt linkStyle default stroke:#4B5563,stroke-width:1px ``` With `on_eval: []`, a failed check records `needs_review` and emits `execution.needs_review`; your app can use that webhook to route a person to the run. If the review record is unavailable or altered, the check gives no verdict. The steps below show the exact fields and commands. ## Example review question A coding worker changes an authorization rule and commits SHA `aaaaaaaa…aaaa` (40 characters). The trusted runner records `team/repo`, base `main`, the full SHA, CodeRabbit CLI version, process result, and JSONL events. The output check asks: **Did CodeRabbit report a completed review of that exact SHA under the selected severity rule?** | Evidence | Hard rule in this example | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Repository, base and source | Match your configured repository/base and the full SHA in the saved review context. | | Completion | One final `complete` event whose status is `review_completed`, with a successful outcome and process exit. | | Unreviewed files | Reject when CodeRabbit reports one or more unreviewed files. If the count is absent, accept only a `completed` outcome; a warning outcome without the count gives no verdict. | | Findings | When the final event reports a finding count, it must match the finding events; no `major` or `critical` finding. | | Freshness | The saved record is at most 24 hours old. | **What this check can say:** CodeRabbit reported that the selected review of the pinned commit against your configured base completed, with no reported major or critical finding and no positive unreviewed-file count. CodeRabbit CLI 0.8.1 can omit that count even on a successful review, so this check cannot prove that every changed file was inspected. It also cannot say the code has no defects or that the commit answers this particular Millwork task. For example, a completed review with one major finding returns: ```json theme={null} { "is_correct": false, "quality_score": 0, "anchor_results": { "evidence_found": true, "repository_matches": true, "base_matches": true, "sha_matches": true, "review_completed": true, "no_reported_unreviewed_files": true, "severity_allowed": false, "capture_fresh": true } } ``` `severity_allowed: false` names the failed rule. `no_reported_unreviewed_files: true` means CodeRabbit did not report a positive unreviewed-file count; the count may be absent, so it is not proof of file-by-file coverage. With `on_eval: []`, Millwork records `needs_review` and emits `execution.needs_review`. A failed CodeRabbit process, altered record, or unavailable store yields **no verdict**. `quality_score` cannot override the hard result and is not a code-quality grade. ## Before you begin * Node.js 20 or newer and the published `@millwork/solver@0.1.19` CLI. [Install that pinned version](/get-started/tenant-start#run-your-next-task), or ask your administrator to install it if your organization manages CLI access. Recipe E uses the existing `--recipe d`; there is no `--recipe e` command. The live API steps also use `curl` and `jq`. * For the local cases, no API key or CodeRabbit account is needed. Downloading the checked example bundle requires one docs-site network request; after that, the cases run locally without CodeRabbit or Millwork API calls. * For the live path, a clean checkout at a full commit SHA, an installed GitHub Cloud repository in your CodeRabbit organization, CodeRabbit CLI 0.7.7 or newer with authorized authentication, and customer-controlled private evidence storage. [CodeRabbit's CLI reference](https://docs.coderabbit.ai/cli/reference) describes remote `--agent` review and its completion fields. Your plan may limit review rate. * A fast HTTPS agent endpoint and output-check endpoint. The agent endpoint must respond within 30 seconds, the output check within 10 seconds, and its connection probe within 3 seconds. Finish code work and CodeRabbit review before either call. After installation, check the version used by this recipe: ```bash theme={null} millwork --version --json ``` **Expected:** `package_version` is `0.1.19`. Keep that version for the protected-check connection steps below. Start with Recipe D's maintained adapter. Its first local cases need no keys: ```bash theme={null} millwork verifier init review-check --recipe d --json cd review-check millwork verifier test --local --check selected-check.mjs --access authenticated --json ``` **Expected:** `selected_recipe` is `d` and the local report has `passed: true`. If the CLI does not recognize `--recipe`, check its installed version before continuing. [Recipe D](/cookbook/output-checks/completion-evidence) explains the trusted evidence pattern you will adapt. The checked example bundle below contains the saved-review reader, capture command, fast pointer endpoint, fixture seeder, and negative cases. While still in `review-check`, copy the downloader into `coderabbit-example.mjs`, review it, then run it. It creates a new `coderabbit-review-check` subdirectory, downloads only this docs site's example asset, and checks every file and the bundle manifest before writing anything. ```javascript theme={null} // Generated from the CodeRabbit review-check example. Review before running. import { createHash } from "node:crypto"; import { existsSync, mkdirSync, writeFileSync } from "node:fs"; import { dirname, resolve, sep } from "node:path"; const target = resolve("coderabbit-review-check"); const url = "https://docs.getmillwork.dev/downloads/coderabbit-review-check.json"; const expected = "c1098485a34c0cab185886ff55121ddebf626b596ddf8ecff16b66dc42454a47"; const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex"); if (existsSync(target)) throw new Error("Target directory already exists"); const response = await fetch(url); if (!response.ok) throw new Error(`Example download failed (HTTP ${response.status})`); const bytes = Buffer.from(await response.arrayBuffer()); if (bytes.length > 100_000) throw new Error("Example bundle is too large"); const bundle = JSON.parse(bytes.toString("utf8")); if (!Array.isArray(bundle.files) || bundle.files.length !== 11) throw new Error("Unexpected file inventory"); const seen = new Set(); const contents = []; for (const { path: name, sha256: digest, content } of bundle.files) { if (typeof name !== "string" || !name || name.startsWith("/") || name.includes("\\") || name.split("/").some((part) => !part || part === "." || part === "..") || seen.has(name)) throw new Error("Unsafe file path"); if (typeof digest !== "string" || !/^[0-9a-f]{64}$/.test(digest) || typeof content !== "string") throw new Error("Invalid file entry"); const path = resolve(target, name); if (!path.startsWith(target + sep)) throw new Error("Unsafe target path"); const data = Buffer.from(content, "utf8"); if (sha256(data) !== digest) throw new Error(`Checksum mismatch: ${name}`); seen.add(name); contents.push({ name, digest, path, data }); } const actual = sha256(contents.map(({ name, digest }) => `${name}\0${digest}\n`).join("")); if (actual !== expected || bundle.aggregate_sha256 !== expected) throw new Error("Bundle manifest mismatch"); for (const { path, data } of contents) { mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, data, { flag: "wx" }); } process.stdout.write(`Created ${target} with ${contents.length} checked files.\n`); ``` ```bash theme={null} node coderabbit-example.mjs cd coderabbit-review-check npm run cases npm run seed CR_EVIDENCE_DIR="$PWD/offline-store" \ CR_EVIDENCE_KEY="offline-only-secret-key-at-least-32-characters" \ CR_REPO="team/repo" CR_BASE="main" \ millwork verifier test --local --check check.mjs --access authenticated --json ``` The displayed key is synthetic fixture data, never a production credential. **Expected:** the Node cases pass, then the published CLI reports `passed: true` for its contract cases. Its labelled cases show a completed review, major finding, skipped review, reported unreviewed file, wrong SHA and stale record. The Node suite also checks a dated raw CodeRabbit CLI 0.8.1 capture that omitted the optional count, plus tampering and an unavailable store. The passing seeded fixture expires after 24 hours; remove your own `offline-store` directory and run `npm run seed` again if it has expired. No command here contacts CodeRabbit, TypeSafe or a paid Millwork run. To use this check in the Recipe D adapter you generated, return to `review-check`, place the two check files beside `selected-check.mjs`, and replace that generated selection file: ```bash theme={null} cd .. cp coderabbit-review-check/check.mjs coderabbit-review-check/review-record.mjs . cat > selected-check.mjs <<'EOF' export { runHardCheck, scoreQuality, labelledCases } from "./check.mjs"; EOF CR_EVIDENCE_DIR="$PWD/coderabbit-review-check/offline-store" \ CR_EVIDENCE_KEY="offline-only-secret-key-at-least-32-characters" \ CR_REPO="team/repo" CR_BASE="main" \ millwork verifier test --local --check selected-check.mjs --access authenticated --json ``` **Expected:** `passed: true`. The generated `existing-node-app.mjs` imports this `selected-check.mjs`; keep `check.mjs` and `review-record.mjs` beside it when deploying. Replace the synthetic key and store with private production values before deployment. Your coding worker or CI commits the change first. In a clean checkout of that commit, use the included `capture.mjs` under your runner's control. It resolves the full `git rev-parse HEAD`, calls CodeRabbit's remote review with `--source-branch` set to that SHA, records the CLI version and JSONL, then stores an immutable, signed record. Configure `CR_REPO`, `CR_BASE`, `CR_EVIDENCE_DIR` and `CR_EVIDENCE_KEY` privately; do not put keys in a command argument, chat, task body, or receipt. ```text theme={null} cr review --remote owner/repo --base main --source-branch --agent ``` That is the command the runner executes; the example runs it only when you explicitly launch `capture.mjs`. **Before a live launch**, approve the CodeRabbit destination, review quota or spend limit, and evidence retention with the account owner. A repaired commit needs a new SHA, a new review, and a new evidence ID. Preserve one dated raw `--agent` capture with its exact CLI version for the field mapping you deploy. Do not infer PR-side `pre_merge_checks` from this CLI stream. After that approval, load `CR_REPO`, `CR_BASE`, `CR_EVIDENCE_DIR` and `CR_EVIDENCE_KEY` privately in the clean checkout and run: ```bash theme={null} node /path/to/review-check/coderabbit-review-check/capture.mjs ``` **Expected:** one JSON line with `evidence_id`, the full `reviewed_sha`, and `cli_version`. Keep the evidence ID for the run; a nonzero exit means there is no saved review to use. Keep the evidence store and signing key in your environment. The included `agent-pointer.mjs` reads `task.inputs_ref.json.evidence_id` and returns only: ```json theme={null} { "candidate": { "evidence_id": "cr-" }, "usage": { "usd": 0 } } ``` 1. Deploy the generated `existing-node-app.mjs` with the check files you selected above. Follow [Recipe 0's deployment steps](/cookbook/output-checks/build-the-dock#deploy-and-test-https) to serve it over HTTPS, accept chunked POST requests, and answer the reserved probe without reading the store. Its [endpoint contract](/cookbook/output-checks/build-the-dock#the-endpoint-contract) defines the callback. 2. Set `CR_AGENT_TOKEN`, `CR_EVIDENCE_DIR` and `CR_EVIDENCE_KEY` privately. Start the sample pointer process with `node /path/to/review-check/coderabbit-review-check/agent-pointer.mjs`. It listens on localhost by default. 3. Place the pointer process behind your HTTPS ingress. Protect both endpoints with private bearer credentials. Keep the check's key and raw review in your environment; only the opaque evidence ID enters the Millwork candidate. 4. Before sending test traffic, get the endpoint owner's approval. Then test protected access before a live run. The agent endpoint can read the pointer supplied in the task, but that pointer is **not** trusted review proof. The check reads its own signed record and compares the saved review context with your configured repository and base. When CodeRabbit reports `remote: true` rather than a repository name, repository identity rests on the trusted runner's `CR_REPO` value. The check receives no run ID or expected commit SHA, so it cannot prove this review belongs to the current task. Ask your administrator to register the agent endpoint's bearer token and give you its credential handle. Put that handle in `endpoint.auth_ref`, never the token itself. Save the HTTPS endpoint as an agent option with `POST /v1/arms`, setting `data_class_grants` to a class your organization supports. Keep the returned `arm_id`. If your organization cannot provision an agent credential handle yet, finish the offline path and ask its administrator to set up the broker before a live run. ```json theme={null} { "kind": "agent", "display_name": "Completed CodeRabbit review pointer", "endpoint": { "url": "https://agents.example.com/coderabbit-pointer", "auth_ref": "" }, "capability_tags": ["completed_code_review"], "data_class_grants": ["sandbox"] } ``` The endpoint URL must be your deployed agent endpoint; it receives `{ "task": { ... } }` and returns the saved pointer. The handle must resolve to the same bearer token you configured for that endpoint. The example's `sandbox` grant must be available to your organization. Save that body as `coderabbit-agent.json`. After its owner approves the agent endpoint's registration probe, use an organization API key to register it. The sample endpoint answers the authenticated `HEAD` probe without reading evidence or starting a review: ```bash theme={null} curl --fail-with-body --request POST \ --header "Authorization: Bearer $SOLVERAPI_API_KEY" \ --header "Content-Type: application/json" \ --data @coderabbit-agent.json \ --output coderabbit-agent-response.json \ "https://api.getmillwork.dev/v1/arms" && jq '{arm_id, status, status_reason, preflight}' coderabbit-agent-response.json ``` **Expected:** HTTP 201 and `status: "ready"`; keep `arm_id`. If the response is lost, list `GET /v1/arms` before creating another option. A degraded status means the endpoint or brokered credential needs repair before the run. Connect the **output-check endpoint** separately with the same pinned CLI 0.1.19. The CLI reads the `SOLVERAPI_API_KEY` used for the API call above; keep it in your private terminal environment. Ask the endpoint owner to approve the registration probe. At that endpoint, issue a dedicated adapter key and configure it to accept the key. Set `VERIFIER_URL` to its deployed HTTPS route; `CUSTOMER_APP_ORIGIN` is the Millwork app address for private key entry, not the check endpoint: ```bash theme={null} export VERIFIER_URL="https://checks.example.com/coderabbit-review" export CUSTOMER_APP_ORIGIN="https://app.getmillwork.dev" millwork verifier connect \ --endpoint "$VERIFIER_URL" \ --access managed \ --name "CodeRabbit review check" \ --version "1.0.0" \ --stop-days 90 \ --connect-only ``` Enter the adapter key only in the hidden terminal prompt. Customer-controlled automation may instead supply `VERIFIER_CONNECTION_SECRET` in the command's host environment; never place the key in a command argument, chat, or task body. Without a private interactive terminal or configured secret, the command returns `state: "action_required"` with a saved `verifier_id`, private `continue_url`, `intent_id`, and expiry. The intended person opens that URL in their browser, signs in to the same organization, and enters the key privately. Then finish the saved intent using the same organization API key: ```bash theme={null} millwork verifier continue \ --verifier-id "" \ --intent-id "" \ --json ``` Keep the returned `verifier_id` only after `connection.status` is `active` and the ID matches the saved intent. A pending ID alone is not a connected check. Set `VERIFIER_ID` to that ID and test the protected route with a fresh request key before selecting it for a run: ```bash theme={null} export VERIFIER_ID="" TEST_REQUEST_KEY="verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')" millwork verifier test \ --verifier-id "$VERIFIER_ID" \ --idempotency-key "$TEST_REQUEST_KEY" \ --json ``` **Expected:** `headline: "ready"`, `probe.contract.validated: true`, and the same `verifier_id`. The reserved probe may return `is_correct: false` on purpose; that does not make a compatible endpoint unavailable. If connection or probe fails, repair it before the run. Keep `arm_id` and `VERIFIER_ID` for the request below. In the run request, pin `routing.required_arm_id`, send the opaque evidence ID in `task.inputs_ref.json`, set the approved budget, select `verifier_id`, and use `on_eval: []`. Review the full request and get explicit approval before sending a paid run. [Results and receipts](/concepts/results-and-receipts) covers the recorded outcome. ```json theme={null} { "mode": "live", "task": { "objective": "Check the completed authorization change", "inputs_ref": { "kind": "inline_json", "json": { "evidence_id": "cr-" } } }, "policy": { "data_classes": ["sandbox"], "budget": { "max_cost_usd": 0.05, "max_runtime_s": 30 }, "on_eval": [] }, "routing": { "required_arm_id": "" }, "verifier_id": "" } ``` The request sends only an opaque evidence ID, not source code or raw review text. Use a data class your organization and saved agent actually support; the current API accepts `public` and `sandbox` for saved arms. Save the JSON above as `coderabbit-run.json`. Only after the run owner approves that exact request, set a unique `RUN_KEY` and submit it with your organization API key: ```bash theme={null} RUN_KEY="" curl --fail-with-body --request POST \ --header "Authorization: Bearer $SOLVERAPI_API_KEY" \ --header "Idempotency-Key: $RUN_KEY" \ --header "Content-Type: application/json" \ --data @coderabbit-run.json \ --output coderabbit-execution.json \ "https://api.getmillwork.dev/v1/executions" && jq '{execution_id, status, mode}' coderabbit-execution.json ``` **Expected:** the submission returns a queued execution ID, not a completed result. Read only the status and receipt for this pointer-only agent run: ```bash theme={null} EXECUTION_ID="$(jq -er '.execution_id' coderabbit-execution.json)" curl --fail-with-body \ --header "Authorization: Bearer $SOLVERAPI_API_KEY" \ --output coderabbit-status.json \ "https://api.getmillwork.dev/v1/executions/$EXECUTION_ID" && jq '{execution_id, status}' coderabbit-status.json ``` While status is `queued`, `running`, or `progress`, repeat only the status read; do not submit the run again. Once status is final, fetch its receipt: ```bash theme={null} curl --fail-with-body \ --header "Authorization: Bearer $SOLVERAPI_API_KEY" \ --output coderabbit-receipt.json \ "https://api.getmillwork.dev/v1/receipts/$EXECUTION_ID" && jq '{execution_id, status, totals, check: .slices[0].verifier}' coderabbit-receipt.json ``` The [results and receipts guide](/concepts/results-and-receipts) explains the fields. A complete allowed review has `is_correct: true` and named receipt results. A hard false with `on_eval: []` becomes `needs_review` and emits `execution.needs_review`; it does not start fallback or repair. A technical outage produces no review verdict. The receipt records the Millwork platform fee separately from agent-reported usage; the earlier CodeRabbit review cost is external and is not included. ## Interpret and recover | What you see | What it means | Next action | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `is_correct: true` with all named results true | The selected pinned review met this check's policy. | Continue under your release policy; this is not PR merge approval. | | `severity_allowed: false` | A major or critical finding was recorded. | Fix the code, create a new SHA, request a new CodeRabbit review, and submit a new evidence ID. | | `review_completed: false` or `no_reported_unreviewed_files: false` | The review was skipped or CodeRabbit reported an unreviewed file. | Complete the review or adjust its scope deliberately; do not reuse an incomplete result. | | `sha_matches: false` or `capture_fresh: false` | The reviewed SHA differs or the record is too old. | Check your runner's pinned commit and capture a fresh review. | | A recent evidence ID used for another task | The same signed record can be reused during its 24-hour validity window. | Bind the record to the intended task in your own runner before submitting a run; do not treat this check as task-binding proof. | | No verdict, endpoint error or timeout | The review process, store, record integrity or endpoint failed; a warning outcome without an unreviewed-file count is also ambiguous. | Repair the source; do not turn an outage into a pass or automatic paid replay. | The receipt shows whether the saved CodeRabbit review met these rules. It cannot show whether that commit completes this task: the check cannot compare the reviewed SHA with this task's expected SHA. A signed review record can be reused for another task within 24 hours. Bind each record to the intended task in your own runner before submitting a run. Do not treat the receipt as a code-safety certificate.
Keep the review runner, store, agent endpoint and check in your own environment. Use the offline matrix first. Before any vendor review or paid Millwork run, approve its destination, budget and retention. Record the exact CLI version and raw capture used to confirm the field mapping. Inspect installed Millwork and CodeRabbit versions. Run Recipe D and the downloaded offline cases. Report the full SHA, base, selected scope, local results, and any unsupported field or command. Keep credentials out of chat. Stop before vendor calls, endpoint test traffic, or a paid run until the responsible person authorizes each step. After an approved run, report the named receipt results and the next recovery action.
For a separate semantic question about source meaning, use the existing [Jev recipe](/cookbook/output-checks/jev-citation-check). The exact SHA, review-completion and severity rules here remain code-owned; a Jev score cannot establish them.