Run a coding agent with review evidence

Goal: start a coding task, let Millwork register its final commit and open a pull request, then read one decision packet that ties GitHub and CodeRabbit evidence to that commit. You are done when: your first run has a decision_packet with the pull request, full final SHA, each source’s result, and a final verdict. For the next task, get the current base SHA and use a new request key. This path is for a coding agent that can work asynchronously. Millwork runs the review coordination and stores the evidence decision. Your agent still runs on your chosen host. The maintained Node adapter connects a Codex CLI or another configured agent to Millwork’s async_job_v1 protocol; it does not run CodeRabbit CLI or keep a separate review evidence store.

Creating the run branch and PR can start your own CI and CodeRabbit usage. The Review Composer fee defaults to $0.10 per accepted run, instead of the ordinary run fee. Your max_cost_usd must cover at least that fee. Check your configured fee in GET /v1/account at billing.review_composer_fee_usd_per_execution; it can differ from the default. Millwork charges the fee in addition to any coding-agent provider, GitHub, or CodeRabbit charges on your accounts. Set provider spending limits in those accounts before a live run.

What the packet decides

Millwork reads the run branch through its GitHub App and registers its current head SHA. It asks the configured sources about that same SHA. With combine: "all", each source must pass. With combine: "any", one pass can suffice unless a veto source blocks or is unavailable. A missing or stale review gives no_verdict; it is never assumed to be a pass. A changed commit requires new evidence. The packet is a record of the selected evidence and policy. It does not certify that the code has no defects or override your repository’s branch protection.

Connect the repository

Open Settings → Integrations. Under Connected repositories, select Add repositories. GitHub opens the Millwork GitHub App page. Choose the personal account or GitHub organization that owns the repository. Choose Only select repositories and pick the repositories Millwork needs. Select Install, or Save if the App is already installed. GitHub may ask you to authorize Millwork for your GitHub account, then returns you to Millwork.Millwork shows Repositories updated and updates the connected repository list. Every repository you selected and can push to shows Connected under your GitHub account. A repository you removed from the App’s access leaves the list. Millwork never asks you for a repository name.The other results tell you what to do next:

Waiting for approval: a GitHub organization owner needs to approve the Millwork GitHub App. After they approve, select Add repositories again and finish on GitHub. Your repositories appear once Millwork confirms your access.
No repository to connect: GitHub granted no repository you can push to. Select Add repositories and include one you can write to.
GitHub did not confirm the change: select Add repositories again.
App access missing on a row: the repository was removed from the App on GitHub. Select Add repositories to grant it again.
App status unavailable on a row: Millwork could not reach GitHub to check it. Select Sync repositories to check again.

After changing the App’s repository access directly on GitHub for an existing connection, select Sync repositories to update the list. If no repository is connected yet, select Add repositories instead.To change repository access, open a repository’s row menu and choose Manage repo…. GitHub opens in a new tab, where you can add, remove, or swap repositories. GitHub won’t remove the last selected repository: add another first, or uninstall the App to remove all access. After saving, Millwork refreshes your original Integrations tab and closes the GitHub tab. If the original tab is gone, continue in the returned tab. Select Sync repositories if the list has not refreshed.Only the person who connected a GitHub account can choose Revoke GitHub authorization… for it. Other members who can manage integrations still see Manage repo…; GitHub controls which repository changes they can make. Millwork checks the account with GitHub before revoking. Revoking also affects this account’s connections in other Millwork organizations; those organizations will need to reconnect it. In this organization, its connected repositories show Reconnect required. Repositories connected by other GitHub accounts stay connected. The App stays installed with its repository access. If GitHub cannot confirm the revoke, Millwork shows an error and keeps the last confirmed connection state; refresh before trying again.Millwork opens the run’s pull request (PR) in the connected GitHub account’s name. The App keeps access to the run branch and its checks and reviews. Use the exact owner/repo shown under Connected repositories for later requests. If a row shows Reconnect required, open that row’s menu and select Reconnect on GitHub before starting another run.For a private repository, use tenant_internal in the request. Review Composer accepts it only for a connected repository and registered agent. Other run and output check paths continue to refuse it.

Prepare CodeRabbit and your agent

Install CodeRabbit’s GitHub App on the same repository. Commit this setting in the base branch’s .coderabbit.yaml before the first run:

reviews:
  request_changes_workflow: true

This is the documented CodeRabbit setting for an approval after comments are resolved, the latest commit is reviewed, and no checks CodeRabbit treats as blocking remain. Millwork requires the CodeRabbit Bot’s substantive APPROVED review on the registered SHA and no unresolved threads started by that bot. A comment or an earlier-SHA approval does not pass.CodeRabbit may show advisory warnings, such as a generic PR title, even after approval. Inspect those warnings in CodeRabbit before merging; Millwork does not parse their text into the decision packet.For a private repository, confirm the connected GitHub account has a CodeRabbit seat and that CodeRabbit is enabled on this repository. If CodeRabbit does not start, inspect its repository configuration and plan limits; a bot-posted review command is not proof of an automatic review.Deploy an async_job_v1 agent endpoint. Copy the maintained server.mjs adapter from its setup page; that page also covers the protocol, stable retry identity, non-force push, cancellation and restart behavior. Generate a random token of at least 32 characters and configure it as REVIEW_ADAPTER_TOKEN on the adapter host. Keep it in your secret manager. The endpoint must answer authenticated HEAD with 200 and accept POST start, status and cancel at the same HTTPS URL.If you supply another async_job_v1 adapter, persist the job identity before starting work. Return job not found for a stable attempt key only if no start was ever accepted for that key. After acceptance, status must find that same job across process restarts, including a terminal failure. Millwork can safely reissue start with the original key only on an explicit job not found response.In Settings → Integrations, select Register agent under Coding agents to register this endpoint. Select Enter key and save the key on the private page. Return to Integrations and select Check for entered key if offered. When the row says Key ready to test, open its row menu and select Test and connect. The dashboard never asks for the token in the registration form. The equivalent dedicated API route checks that the named repository is already connected and sets the private grant and protocol fields for you:

{
  "repository": "team/repo",
  "display_name": "My coding agent",
  "endpoint": {
    "url": "https://agents.example.com/millwork-review",
    "auth_ref": ""
  },
  "max_runtime_s": 3600
}

Send it to POST /v1/review-composer/agents with your Millwork organization key. It returns an arm_id (your agent ID) with status: "degraded" until the token is privately connected. The TypeScript SDK exposes solver.reviewComposerAgents.create(body, { idempotencyKey }).Start the private handoff with POST /v1/review-composer/agents/{arm_id}/connection/intents and {"stop_choice":{"kind":"preset_days","days":90}}, or call solver.reviewComposerAgents.createConnectionIntent(armId, { kind: "preset_days", days: 90 }). Open its continue_url in a signed-in browser. Enter the same token you set on the adapter host. The link expires after 15 minutes and can be used once. The token goes to Millwork’s separate agent custody; it is never part of agent registration, run submission, or the receipt.After saving the token, call GET /v1/review-composer/agents/{arm_id}/connection. Copy pending_key.handle and pending_key.captured_generation into POST /v1/review-composer/agents/{arm_id}/connection/test. The test sends an authenticated HEAD to your registered endpoint. Your agent becomes ready only when the endpoint returns 200. In the SDK, use connection(armId) and testConnection(armId, pending_key). You can also inspect, replace, or revoke the token through these connection routes; a revoked or expired token cannot authorize another run. Keep the agent ID once status is ready.

Choose the GitHub rule

Choose one or more required GitHub Actions jobs by workflow ID and job name, not by a loose check label. In an authenticated GitHub CLI session, list workflow IDs with gh api repos/team/repo/actions/workflows --jq '.workflows[] | [.id, .name]'. Open a successful run and copy the job name exactly. Millwork examines all current attempts for the selected workflow and final SHA; a pending rerun or failed job cannot inherit an earlier pass.Get the current full base commit SHA just before submission. For main:

gh api repos/team/repo/git/ref/heads/main --jq '.object.sha'

Use refs/heads/main as base_ref and that 40-character value as base_sha. If the base changes before acceptance, refresh it and create a new request identity. A normal merge that moves the base forward during a run is checked for ancestry; a rewrite cannot silently pass.

Start one run

The default first-use policy asks for both CodeRabbit and one named GitHub job. all means both must pass. on_eval: ["gate"] keeps the action tied to the registered SHA. Set a runtime deadline and a max_cost_usd budget of at least the configured Review Composer fee (default $0.10). The maintained adapter cannot measure or cap the Codex CLI’s separate provider bill; set any provider spending limit in that account. The provider bill is on top of the Millwork fee. Review GitHub and CodeRabbit plan usage before starting.

{
  "mode": "live",
  "task": { "objective": "Add a regression test for the authorization rule and fix the bug." },
  "policy": {
    "data_classes": ["tenant_internal"],
    "budget": { "max_cost_usd": 5, "max_runtime_s": 3600 },
    "on_eval": ["gate"]
  },
  "review_composer": {
    "version": "v1",
    "repository": "team/repo",
    "base_ref": "refs/heads/main",
    "base_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "agent_arm_id": "<ready-agent-arm-id>",
    "sources": [
      { "id": "coderabbit" },
      { "id": "github", "checks": [{ "workflow_id": 123456, "job_name": "test" }], "minimum_approvals": 0 }
    ],
    "combine": "all",
    "veto_sources": []
  }
}

Replace the repository, base SHA, arm ID, workflow ID and job name with your verified values. Send this body to POST /v1/executions with an Idempotency-Key you keep with the request. If the response is uncertain, retry the exact same body and key; do not start another run just to recover the first. In TypeScript, use solver.executions.create(body, { idempotencyKey }). The current public execution MCP does not yet support this option; use REST or the TypeScript SDK for this run.

Inspect the packet and run again

Poll GET /v1/executions/{execution_id} until the run is terminal. Read decision_packet there and in the receipt export. This example shows a GitHub job that passed while CodeRabbit approval remained unavailable; actual evidence references and reasons come from your run:

{
  "version": "v1",
  "verdict": "no_verdict",
  "reason": "required_source_unavailable",
  "artifact": {
    "repository": "team/repo",
    "base_ref": "refs/heads/main",
    "base_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "branch": "refs/heads/millwork/run/exe_example",
    "final_sha": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
    "pr_number": 42,
    "pr_url": "https://github.com/team/repo/pull/42",
    "pr_opened_by": { "github_user_id": 12345, "github_login": "connected-person" }
  },
  "sources": [
    { "id": "coderabbit", "status": "no_verdict", "reason": "review_incomplete", "evidence_id": null, "observed_at": "2026-09-25T12:00:00Z" },
    { "id": "github", "status": "pass", "reason": "github_policy_passed", "evidence_id": "<evidence-id>", "observed_at": "2026-09-25T12:00:00Z" }
  ],
  "policy": { "combine": "all", "veto_sources": [], "sources": ["coderabbit", "github"] },
  "policy_hash": "sha256:<policy-snapshot-digest>",
  "connector_versions": { "github": "github-app/v1", "coderabbit": "github-pr/v1", "agent": "async_job_v1" },
  "action": "needs_review",
  "agent_usage_usd": 0,
  "platform_fee_usd": 0.10,
  "decided_at": "2026-09-25T12:00:00Z"
}

Act on decision_packet.verdict and decision_packet.action, not the execution’s lifecycle status: completed can mean observation ended with review needed. Read each source even if the aggregate passed under any. If the result is block or no_verdict, inspect the source reason, PR and current SHA. Resolve comments, repair CI, or correct permissions as appropriate. A repaired commit must receive fresh checks and review before it can pass. A cancelled, timed-out, or failed review job gives no_verdict rather than approval. agent_usage_usd: 0 means the adapter did not report provider usage; check the provider account for its bill. For the next task, reuse the connected repository, registered agent, and policy shape; choose a fresh base SHA and new request key.

Choose the right CodeRabbit path

Your starting point	Use
Start a coding agent, open a PR and coordinate GitHub plus CodeRabbit evidence	This Review Composer guide and the GitHub + CodeRabbit recipe. Millwork owns the review decision packet.
A separate runner already has a completed CodeRabbit CLI review of a pinned commit	Recipe E: check a CodeRabbit review. Your runner and output check own the signed review record.

Recover without a second agent attempt

What you see	Next step
Submit response lost	Repeat the same request with the same `Idempotency-Key`, or inspect the known execution ID.
Agent or adapter unavailable	Check the endpoint and its saved job ID. Millwork checks that ID after an interruption before deciding what happened; it does not automatically start a second agent job.
CodeRabbit has no current approval	Open the packet’s PR, resolve its threads and request/recheck review of the final SHA. If the run ended with `no_verdict`, start a new run for fresh evidence.
Required GitHub job failed or is still running	Open that workflow’s current run. A failed job blocks; an incomplete or ambiguous run gives no verdict.
Branch moved after registration	Inspect the PR’s new head. The packet records the changed commit; start a new run for it.

Get started

Provide models

Build

Build safely

SDK and MCP

Run a coding agent with review evidence

What the packet decides

Choose the right CodeRabbit path

Recover without a second agent attempt