Check a CodeRabbit review

Goal: check a completed CodeRabbit review of a specific commit in a Millwork run. You are done with the offline example when: its pass, reject and no-verdict cases work. After an approved live run, read the receipt for the final decision. Use this when CodeRabbit already reviews your changes and you want its result to inform a Millwork run. Your output check applies rules you choose, and the run receipt records what happened. If your pull request check already makes the same decision and you need no Millwork run, keep that workflow.

Start with the offline example

Run Recipe D, download the checked example once, then try its local cases without keys or CodeRabbit calls.

How it works

Suppose a coding worker changes an authorization rule. CodeRabbit reviews the committed change before the Millwork run. During the run, your agent endpoint returns an evidence ID; the saved review stays in your store. This diagram shows how the review becomes one output-check result. With on_eval: [], a failed check records needs_review and emits execution.needs_review; your app can use that webhook to route a person to the run. If the review record is unavailable or altered, the check gives no verdict. The steps below show the exact fields and commands.

Example review question

A coding worker changes an authorization rule and commits SHA aaaaaaaa…aaaa (40 characters). The trusted runner records team/repo, base main, the full SHA, CodeRabbit CLI version, process result, and JSONL events. The output check asks: Did CodeRabbit report a completed review of that exact SHA under the selected severity rule?

Evidence	Hard rule in this example
Repository, base and source	Match your configured repository/base and the full SHA in the saved review context.
Completion	One final `complete` event whose status is `review_completed`, with a successful outcome and process exit.
Unreviewed files	Reject when CodeRabbit reports one or more unreviewed files. If the count is absent, accept only a `completed` outcome; a warning outcome without the count gives no verdict.
Findings	When the final event reports a finding count, it must match the finding events; no `major` or `critical` finding.
Freshness	The saved record is at most 24 hours old.

What this check can say: CodeRabbit reported that the selected review of the pinned commit against your configured base completed, with no reported major or critical finding and no positive unreviewed-file count. CodeRabbit CLI 0.8.1 can omit that count even on a successful review, so this check cannot prove that every changed file was inspected. It also cannot say the code has no defects or that the commit answers this particular Millwork task.

For example, a completed review with one major finding returns:

{
  "is_correct": false,
  "quality_score": 0,
  "anchor_results": {
    "evidence_found": true,
    "repository_matches": true,
    "base_matches": true,
    "sha_matches": true,
    "review_completed": true,
    "no_reported_unreviewed_files": true,
    "severity_allowed": false,
    "capture_fresh": true
  }
}

severity_allowed: false names the failed rule. no_reported_unreviewed_files: true means CodeRabbit did not report a positive unreviewed-file count; the count may be absent, so it is not proof of file-by-file coverage. With on_eval: [], Millwork records needs_review and emits execution.needs_review. A failed CodeRabbit process, altered record, or unavailable store yields no verdict. quality_score cannot override the hard result and is not a code-quality grade.

Before you begin

Node.js 20 or newer and the published @millwork/solver@0.1.19 CLI. Install that pinned version, or ask your administrator to install it if your organization manages CLI access. Recipe E uses the existing --recipe d; there is no --recipe e command. The live API steps also use curl and jq.
For the local cases, no API key or CodeRabbit account is needed. Downloading the checked example bundle requires one docs-site network request; after that, the cases run locally without CodeRabbit or Millwork API calls.
For the live path, a clean checkout at a full commit SHA, an installed GitHub Cloud repository in your CodeRabbit organization, CodeRabbit CLI 0.7.7 or newer with authorized authentication, and customer-controlled private evidence storage. CodeRabbit’s CLI reference describes remote --agent review and its completion fields. Your plan may limit review rate.
A fast HTTPS agent endpoint and output-check endpoint. The agent endpoint must respond within 30 seconds, the output check within 10 seconds, and its connection probe within 3 seconds. Finish code work and CodeRabbit review before either call.

After installation, check the version used by this recipe:

millwork --version --json

Expected: package_version is 0.1.19. Keep that version for the protected-check connection steps below.

Try the published completion check

Start with Recipe D’s maintained adapter. Its first local cases need no keys:

millwork verifier init review-check --recipe d --json
cd review-check
millwork verifier test --local --check selected-check.mjs --access authenticated --json

Expected: selected_recipe is d and the local report has passed: true. If the CLI does not recognize --recipe, check its installed version before continuing. Recipe D explains the trusted evidence pattern you will adapt.

Run the CodeRabbit adapter offline

The checked example bundle below contains the saved-review reader, capture command, fast pointer endpoint, fixture seeder, and negative cases. While still in review-check, copy the downloader into coderabbit-example.mjs, review it, then run it. It creates a new coderabbit-review-check subdirectory, downloads only this docs site’s example asset, and checks every file and the bundle manifest before writing anything.

Copy the downloader

// Generated from the CodeRabbit review-check example. Review before running.
import { createHash } from "node:crypto";
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
import { dirname, resolve, sep } from "node:path";

const target = resolve("coderabbit-review-check");
const url = "https://docs.getmillwork.dev/downloads/coderabbit-review-check.json";
const expected = "c1098485a34c0cab185886ff55121ddebf626b596ddf8ecff16b66dc42454a47";
const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
if (existsSync(target)) throw new Error("Target directory already exists");
const response = await fetch(url);
if (!response.ok) throw new Error(`Example download failed (HTTP ${response.status})`);
const bytes = Buffer.from(await response.arrayBuffer());
if (bytes.length > 100_000) throw new Error("Example bundle is too large");
const bundle = JSON.parse(bytes.toString("utf8"));
if (!Array.isArray(bundle.files) || bundle.files.length !== 11) throw new Error("Unexpected file inventory");
const seen = new Set();
const contents = [];
for (const { path: name, sha256: digest, content } of bundle.files) {
  if (typeof name !== "string" || !name || name.startsWith("/") || name.includes("\\") || name.split("/").some((part) => !part || part === "." || part === "..") || seen.has(name)) throw new Error("Unsafe file path");
  if (typeof digest !== "string" || !/^[0-9a-f]{64}$/.test(digest) || typeof content !== "string") throw new Error("Invalid file entry");
  const path = resolve(target, name);
  if (!path.startsWith(target + sep)) throw new Error("Unsafe target path");
  const data = Buffer.from(content, "utf8");
  if (sha256(data) !== digest) throw new Error(`Checksum mismatch: ${name}`);
  seen.add(name);
  contents.push({ name, digest, path, data });
}
const actual = sha256(contents.map(({ name, digest }) => `${name}\0${digest}\n`).join(""));
if (actual !== expected || bundle.aggregate_sha256 !== expected) throw new Error("Bundle manifest mismatch");
for (const { path, data } of contents) {
  mkdirSync(dirname(path), { recursive: true });
  writeFileSync(path, data, { flag: "wx" });
}
process.stdout.write(`Created ${target} with ${contents.length} checked files.\n`);

node coderabbit-example.mjs
cd coderabbit-review-check
npm run cases
npm run seed
CR_EVIDENCE_DIR="$PWD/offline-store" \
CR_EVIDENCE_KEY="offline-only-secret-key-at-least-32-characters" \
CR_REPO="team/repo" CR_BASE="main" \
millwork verifier test --local --check check.mjs --access authenticated --json

The displayed key is synthetic fixture data, never a production credential. Expected: the Node cases pass, then the published CLI reports passed: true for its contract cases. Its labelled cases show a completed review, major finding, skipped review, reported unreviewed file, wrong SHA and stale record. The Node suite also checks a dated raw CodeRabbit CLI 0.8.1 capture that omitted the optional count, plus tampering and an unavailable store. The passing seeded fixture expires after 24 hours; remove your own offline-store directory and run npm run seed again if it has expired. No command here contacts CodeRabbit, TypeSafe or a paid Millwork run.To use this check in the Recipe D adapter you generated, return to review-check, place the two check files beside selected-check.mjs, and replace that generated selection file:

cd ..
cp coderabbit-review-check/check.mjs coderabbit-review-check/review-record.mjs .
cat > selected-check.mjs <<'EOF'
export { runHardCheck, scoreQuality, labelledCases } from "./check.mjs";
EOF
CR_EVIDENCE_DIR="$PWD/coderabbit-review-check/offline-store" \
CR_EVIDENCE_KEY="offline-only-secret-key-at-least-32-characters" \
CR_REPO="team/repo" CR_BASE="main" \
millwork verifier test --local --check selected-check.mjs --access authenticated --json

Expected: passed: true. The generated existing-node-app.mjs imports this selected-check.mjs; keep check.mjs and review-record.mjs beside it when deploying. Replace the synthetic key and store with private production values before deployment.

Capture one completed review in your runner

Your coding worker or CI commits the change first. In a clean checkout of that commit, use the included capture.mjs under your runner’s control. It resolves the full git rev-parse HEAD, calls CodeRabbit’s remote review with --source-branch set to that SHA, records the CLI version and JSONL, then stores an immutable, signed record. Configure CR_REPO, CR_BASE, CR_EVIDENCE_DIR and CR_EVIDENCE_KEY privately; do not put keys in a command argument, chat, task body, or receipt.

cr review --remote owner/repo --base main --source-branch <full-40-character-sha> --agent

That is the command the runner executes; the example runs it only when you explicitly launch capture.mjs. Before a live launch, approve the CodeRabbit destination, review quota or spend limit, and evidence retention with the account owner. A repaired commit needs a new SHA, a new review, and a new evidence ID. Preserve one dated raw --agent capture with its exact CLI version for the field mapping you deploy. Do not infer PR-side pre_merge_checks from this CLI stream.After that approval, load CR_REPO, CR_BASE, CR_EVIDENCE_DIR and CR_EVIDENCE_KEY privately in the clean checkout and run:

node /path/to/review-check/coderabbit-review-check/capture.mjs

Expected: one JSON line with evidence_id, the full reviewed_sha, and cli_version. Keep the evidence ID for the run; a nonzero exit means there is no saved review to use.

Connect the two fast endpoints

Keep the evidence store and signing key in your environment. The included agent-pointer.mjs reads task.inputs_ref.json.evidence_id and returns only:

{ "candidate": { "evidence_id": "cr-<opaque-id>" }, "usage": { "usd": 0 } }

Deploy the generated existing-node-app.mjs with the check files you selected above. Follow Recipe 0’s deployment steps to serve it over HTTPS, accept chunked POST requests, and answer the reserved probe without reading the store. Its endpoint contract defines the callback.
Set CR_AGENT_TOKEN, CR_EVIDENCE_DIR and CR_EVIDENCE_KEY privately. Start the sample pointer process with node /path/to/review-check/coderabbit-review-check/agent-pointer.mjs. It listens on localhost by default.
Place the pointer process behind your HTTPS ingress. Protect both endpoints with private bearer credentials. Keep the check’s key and raw review in your environment; only the opaque evidence ID enters the Millwork candidate.
Before sending test traffic, get the endpoint owner’s approval. Then test protected access before a live run.

The agent endpoint can read the pointer supplied in the task, but that pointer is not trusted review proof. The check reads its own signed record and compares the saved review context with your configured repository and base. When CodeRabbit reports remote: true rather than a repository name, repository identity rests on the trusted runner’s CR_REPO value. The check receives no run ID or expected commit SHA, so it cannot prove this review belongs to the current task.

Authorize a run and read the receipt

Ask your administrator to register the agent endpoint’s bearer token and give you its credential handle. Put that handle in endpoint.auth_ref, never the token itself. Save the HTTPS endpoint as an agent option with POST /v1/arms, setting data_class_grants to a class your organization supports. Keep the returned arm_id. If your organization cannot provision an agent credential handle yet, finish the offline path and ask its administrator to set up the broker before a live run.

{
  "kind": "agent",
  "display_name": "Completed CodeRabbit review pointer",
  "endpoint": {
    "url": "https://agents.example.com/coderabbit-pointer",
    "auth_ref": "<registered-agent-credential-handle>"
  },
  "capability_tags": ["completed_code_review"],
  "data_class_grants": ["sandbox"]
}

The endpoint URL must be your deployed agent endpoint; it receives { "task": { ... } } and returns the saved pointer. The handle must resolve to the same bearer token you configured for that endpoint. The example’s sandbox grant must be available to your organization.Save that body as coderabbit-agent.json. After its owner approves the agent endpoint’s registration probe, use an organization API key to register it. The sample endpoint answers the authenticated HEAD probe without reading evidence or starting a review:

curl --fail-with-body --request POST \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --header "Content-Type: application/json" \
  --data @coderabbit-agent.json \
  --output coderabbit-agent-response.json \
  "https://api.getmillwork.dev/v1/arms" &&
  jq '{arm_id, status, status_reason, preflight}' coderabbit-agent-response.json

Expected: HTTP 201 and status: "ready"; keep arm_id. If the response is lost, list GET /v1/arms before creating another option. A degraded status means the endpoint or brokered credential needs repair before the run.Connect the output-check endpoint separately with the same pinned CLI 0.1.19. The CLI reads the SOLVERAPI_API_KEY used for the API call above; keep it in your private terminal environment. Ask the endpoint owner to approve the registration probe. At that endpoint, issue a dedicated adapter key and configure it to accept the key. Set VERIFIER_URL to its deployed HTTPS route; CUSTOMER_APP_ORIGIN is the Millwork app address for private key entry, not the check endpoint:

export VERIFIER_URL="https://checks.example.com/coderabbit-review"
export CUSTOMER_APP_ORIGIN="https://app.getmillwork.dev"
millwork verifier connect \
  --endpoint "$VERIFIER_URL" \
  --access managed \
  --name "CodeRabbit review check" \
  --version "1.0.0" \
  --stop-days 90 \
  --connect-only

Enter the adapter key only in the hidden terminal prompt. Customer-controlled automation may instead supply VERIFIER_CONNECTION_SECRET in the command’s host environment; never place the key in a command argument, chat, or task body. Without a private interactive terminal or configured secret, the command returns state: "action_required" with a saved verifier_id, private continue_url, intent_id, and expiry. The intended person opens that URL in their browser, signs in to the same organization, and enters the key privately. Then finish the saved intent using the same organization API key:

millwork verifier continue \
  --verifier-id "<verifier_id from action_required>" \
  --intent-id "<intent_id from action_required>" \
  --json

Keep the returned verifier_id only after connection.status is active and the ID matches the saved intent. A pending ID alone is not a connected check. Set VERIFIER_ID to that ID and test the protected route with a fresh request key before selecting it for a run:

export VERIFIER_ID="<active check ID>"
TEST_REQUEST_KEY="verifier-test-$(node -e 'process.stdout.write(crypto.randomUUID())')"
millwork verifier test \
  --verifier-id "$VERIFIER_ID" \
  --idempotency-key "$TEST_REQUEST_KEY" \
  --json

Expected: headline: "ready", probe.contract.validated: true, and the same verifier_id. The reserved probe may return is_correct: false on purpose; that does not make a compatible endpoint unavailable. If connection or probe fails, repair it before the run. Keep arm_id and VERIFIER_ID for the request below.In the run request, pin routing.required_arm_id, send the opaque evidence ID in task.inputs_ref.json, set the approved budget, select verifier_id, and use on_eval: []. Review the full request and get explicit approval before sending a paid run. Results and receipts covers the recorded outcome.

{
  "mode": "live",
  "task": {
    "objective": "Check the completed authorization change",
    "inputs_ref": { "kind": "inline_json", "json": { "evidence_id": "cr-<opaque-id>" } }
  },
  "policy": {
    "data_classes": ["sandbox"],
    "budget": { "max_cost_usd": 0.05, "max_runtime_s": 30 },
    "on_eval": []
  },
  "routing": { "required_arm_id": "<saved-agent-id>" },
  "verifier_id": "<connected-output-check-id>"
}

The request sends only an opaque evidence ID, not source code or raw review text. Use a data class your organization and saved agent actually support; the current API accepts public and sandbox for saved arms. Save the JSON above as coderabbit-run.json. Only after the run owner approves that exact request, set a unique RUN_KEY and submit it with your organization API key:

RUN_KEY="<unique key for this approved run>"
curl --fail-with-body --request POST \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --header "Idempotency-Key: $RUN_KEY" \
  --header "Content-Type: application/json" \
  --data @coderabbit-run.json \
  --output coderabbit-execution.json \
  "https://api.getmillwork.dev/v1/executions" &&
  jq '{execution_id, status, mode}' coderabbit-execution.json

Expected: the submission returns a queued execution ID, not a completed result. Read only the status and receipt for this pointer-only agent run:

EXECUTION_ID="$(jq -er '.execution_id' coderabbit-execution.json)"
curl --fail-with-body \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --output coderabbit-status.json \
  "https://api.getmillwork.dev/v1/executions/$EXECUTION_ID" &&
  jq '{execution_id, status}' coderabbit-status.json

While status is queued, running, or progress, repeat only the status read; do not submit the run again. Once status is final, fetch its receipt:

curl --fail-with-body \
  --header "Authorization: Bearer $SOLVERAPI_API_KEY" \
  --output coderabbit-receipt.json \
  "https://api.getmillwork.dev/v1/receipts/$EXECUTION_ID" &&
  jq '{execution_id, status, totals, check: .slices[0].verifier}' coderabbit-receipt.json

The results and receipts guide explains the fields. A complete allowed review has is_correct: true and named receipt results. A hard false with on_eval: [] becomes needs_review and emits execution.needs_review; it does not start fallback or repair. A technical outage produces no review verdict. The receipt records the Millwork platform fee separately from agent-reported usage; the earlier CodeRabbit review cost is external and is not included.

Interpret and recover

What you see	What it means	Next action
`is_correct: true` with all named results true	The selected pinned review met this check’s policy.	Continue under your release policy; this is not PR merge approval.
`severity_allowed: false`	A major or critical finding was recorded.	Fix the code, create a new SHA, request a new CodeRabbit review, and submit a new evidence ID.
`review_completed: false` or `no_reported_unreviewed_files: false`	The review was skipped or CodeRabbit reported an unreviewed file.	Complete the review or adjust its scope deliberately; do not reuse an incomplete result.
`sha_matches: false` or `capture_fresh: false`	The reviewed SHA differs or the record is too old.	Check your runner’s pinned commit and capture a fresh review.
A recent evidence ID used for another task	The same signed record can be reused during its 24-hour validity window.	Bind the record to the intended task in your own runner before submitting a run; do not treat this check as task-binding proof.
No verdict, endpoint error or timeout	The review process, store, record integrity or endpoint failed; a warning outcome without an unreviewed-file count is also ambiguous.	Repair the source; do not turn an outage into a pass or automatic paid replay.

The receipt shows whether the saved CodeRabbit review met these rules. It cannot show whether that commit completes this task: the check cannot compare the reviewed SHA with this task’s expected SHA. A signed review record can be reused for another task within 24 hours. Bind each record to the intended task in your own runner before submitting a run. Do not treat the receipt as a code-safety certificate.

For a developer
For a coding assistant

Keep the review runner, store, agent endpoint and check in your own environment. Use the offline matrix first. Before any vendor review or paid Millwork run, approve its destination, budget and retention. Record the exact CLI version and raw capture used to confirm the field mapping.

For a separate semantic question about source meaning, use the existing Jev recipe. The exact SHA, review-completion and severity rules here remain code-owned; a Jev score cannot establish them.

Output checks

Operations recipes

Check a CodeRabbit review

Start with the offline example

How it works

Example review question

Before you begin

Interpret and recover

Start with the offline example

​How it works

​Example review question

​Before you begin

​Interpret and recover

How it works

Example review question

Before you begin

Interpret and recover