Check citations with Jev

Goal: check whether a cited source supports a claim, using code for the exact quote check and Jev for the meaning. You are done when: the included local cases pass, and one approved Jev call returns a valid check result before you connect the check to Millwork. This recipe starts with the Millwork CLI’s working citation example. It runs offline without an organization key or TypeSafe account. When you are ready, replace its sample judgment with a call to TypeSafe’s Jev model. Temporal is not required.

Tool	What it does here
Millwork CLI	Copies the citation check, adapter, and local tests; later tests the check you deploy.
TypeSafe Jev	Returns a typed judgment: `supports`, `contradicts`, or `says_nothing`, with confidence.
Your code	Confirms the quote exists, validates Jev’s answer, applies your threshold, and returns the check result.

Try the included example

Install the supported Millwork CLI with Node.js 20 or newer, then copy the citation example:

millwork verifier init jev-citation-check --recipe b --json
cd jev-citation-check
millwork verifier test --local --check selected-check.mjs --access authenticated --json

Expected result: the generated selected-check.mjs points to recipe-b-semantic-judgment.mjs. The local report has passed: true because the supported, contradicted, low-confidence, and failure cases match their expected outcomes. The example uses an offline judgment; these commands do not call Jev or start a paid Millwork run.

Connect Jev to the example

The generated check calls askSemanticEngine only after its exact quote check passes. In recipe-b-semantic-judgment.mjs, replace that function with the version below. The existing createSemanticCheck code shares one answer between its hard decision and quality score. selected-check.mjs re-exports the edited file, so the local test and deployed endpoint use the same check.This change makes the local cases send claim and source text to TypeSafe and may incur evaluator cost. Keep the offline result above as your first run; approve a bounded live test with the TypeSafe account owner before continuing.

async function askSemanticEngine(candidate) {
  const apiKey = process.env.TYPESAFE_API_KEY;
  if (!apiKey) throw new Error("TypeSafe API key is not configured");

  const response = await fetch("https://api.typesafe.ai/v1/systemone", {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "jev-latest",
      state: { claim: candidate.claim, source: candidate.source },
      questions: {
        relation: {
          type: "choice",
          instructions: "How does the source relate to the claim?",
          criteria: {
            supports: "The source states or directly implies the claim",
            contradicts: "The source states or implies the opposite",
            says_nothing: "The source does not settle the claim either way",
          },
        },
      },
    }),
    signal: AbortSignal.timeout(8_000),
  });
  if (!response.ok) throw new Error(`TypeSafe request failed (${response.status})`);

  const payload = await response.json();
  const answer = payload?.answers?.relation;
  if (answer?.type !== "choice"
    || !["supports", "contradicts", "says_nothing"].includes(answer.choice)
    || !Number.isFinite(answer.confidence)
    || answer.confidence < 0 || answer.confidence > 1) {
    throw new Error("TypeSafe returned an unsupported answer");
  }
  return { relation: answer.choice, confidence: answer.confidence };
}

This uses TypeSafe’s System One API and Choice question. The generated check keeps its exact quote lookup and acceptance threshold in your code. Jev’s confidence is one input to that rule, not proof that the claim is true.The sample candidate includes its own source text so you can learn the interface. For an application that requires independent evidence, read the source from your own trusted store before asking Jev. Send only the text needed for the judgment. Keep TYPESAFE_API_KEY in your service configuration, never in the candidate, a command argument, or chat.

Make one live test call

With approval for one TypeSafe request, set TYPESAFE_API_KEY privately in your test environment. From jev-citation-check, run one candidate through the edited check:

node --input-type=module <<'JS'
import { runHardCheck } from "./selected-check.mjs";

const candidate = {
  claim: "The policy permits returns within 30 days.",
  quote: "Returns are accepted within 30 days",
  source: "Returns are accepted within 30 days when the item is unused.",
};
console.log(JSON.stringify(await runHardCheck(candidate), null, 2));
JS

Expected result: one Jev request and a check result with is_correct and named anchor_results. The final boolean depends on Jev’s answer and your threshold; do not expect the offline fixture’s exact confidence. A timeout, refusal, or unexpected answer makes this direct runHardCheck command throw without a verdict. The deployed adapter maps that error to a technical failure with no verdict.The full millwork verifier test --local kit repeats labelled candidates for several compatibility checks. After connecting Jev, it can make multiple paid requests and its offline confidence expectations may no longer fit. Replace the sample cases with representative cases and calibrated expectations before running that full kit against Jev. Keep the full evaluator response inside your service.

Use the same pattern elsewhere

The integration point is one typed judgment. Change the question and the code that consumes it for relevance, routing, moderation, or a scored rubric. Keep exact facts, allowed values, and any human-review decision in your own code. TypeSafe’s question types explain when to use Choice, Noul, or Score. The separate Temporal research example uses Jev for narrow source-support and quality judgments inside a long-running workflow. When your check is ready, deploy and connect it. That step requires your HTTPS endpoint and a separate approval for a paid Millwork run. The local example alone needs neither.

Output checks

Operations recipes

Check citations with Jev

Use the same pattern elsewhere

​Use the same pattern elsewhere

Use the same pattern elsewhere