Skip to main content
Goal: add one narrow semantic decision without giving the evaluator control of exact policy or side effects. You are done when: your customized check has a local passed: true report for a supported claim, a rejection, and an unavailable evaluator with no verdict. Use this recipe when code cannot decide a focused property such as citation support, relevance, urgency, policy risk, completeness, or a rubric level. Code should still prove quote presence, numeric limits, and other exact facts. Do not ask one vague question that hides several decisions. Do not turn an unvalidated confidence value into an automatic hard verdict. A supported citation above your confidence threshold passes. A contradiction or low confidence rejects; an unavailable evaluator returns no verdict.
1

Create the example

Expected result: selected_recipe is b, selected_check ends in recipe-b-semantic-judgment.mjs, deployed_check ends in selected-check.mjs, and next_action tests selected-check.mjs.
2

Run the local cases

Run the printed command:
Expected result: passed is true. Check the labelled outcomes if a case fails.
  • labelled.b-pass-supported-citation as a pass;
  • labelled.b-reject-contradicted-citation as a rejection;
  • labelled.b-reject-low-confidence as a rejection; and
  • labelled.b-technical-semantic-service as HTTP 500 with no verdict.
Recovery: if the CLI does not recognize --recipe, stop. Check the install guide for a recipe-capable version; until it lists one, use its non-recipe path. If a labelled case fails, inspect its named results. Fix the exact lookup, judgment mapping, or threshold, then rerun the same local command.
3

Connect your evaluator

The generated askSemanticEngine function is an offline example. Replace it with your server-side client, then keep these boundaries:
  1. send only the claim and relevant source context;
  2. ask one question with supports, contradicts, and says_nothing answers;
  3. check quote presence with ordinary string or span code;
  4. keep the automatic-accept threshold in your code;
  5. reject low-confidence results in this check; if you need human review, route the failed confidence_meets_threshold result in your application;
  6. return a technical failure when the evaluator times out or is unavailable.
The generated example shares one evaluation promise between runHardCheck and scoreQuality. Preserve that sharing so one candidate creates one evaluator call and both outputs use the same answer. Its quality score measures quote presence and a supporting relation, not the evaluator’s confidence. The contradicted example scores at most 0.2.For classification, ask for one allowed label; for detection, one boolean; for scoring, one bounded rubric level; and for routing, one allowed destination ID. Replace the citation-specific anchors with exact checks for that answer. Keep the hard policy and any review routing in your application.Before you send a billable evaluator request, the person who owns that account must approve the exact held-out or endpoint-test batch and its spending limit. Local tests use the offline example and incur no evaluator cost.Do not copy the example confidence threshold. Measure errors and review rates on representative held-out cases, then record who may change the threshold. Rerun the Step 2 command against selected-check.mjs after your edits. Deploy only when your own pass, rejection, and technical-failure cases all pass.
4

Continue to deployment

Read the shared endpoint contract and reserved-probe limits. Your next executable step is Deploy and test HTTPS. Continue with Recipe 0 for connection, paid-run approval, receipt reading, and recovery. In the final receipt for a checked run, confirm quote_present, relation_supports, and confidence_meets_threshold under named results.
Require the assistant to separate exact checks from the evaluator question and to show all three local outcomes. Supply evaluator credentials through your service, never chat. Approve the exact billable test batch before it runs. Approve each paid Millwork run separately after you inspect its preview.