--- title: "Cookbook: Self-consistency — nouls" type: cookbook tags: [cookbook, consistency, noul, evaluation, uncertainty] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__consistency_noul_cookbook.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "Runs a 14-Noul claims rubric 15 times against Jev and six LLM conditions; Jev's mean probability std dev is 0.0102 at 111ms per call, and an uncertain band routes 0.30-0.70 to a human." --- # Cookbook: Self-consistency — nouls > **TL;DR** A repeatability experiment, not a recipe. One auto-insurance claim, a 14-question `Noul` rubric, 15 repeats per condition, across Jev (`jev-latest`, resolved to `jev-1.13.0` in all 15 calls) and six LLM conditions. TypeSafe reports a mean per-question probability standard deviation of `0.0102`, "below all LLM probability conditions here," at 111ms and \$0.000043 per 14-question call. The applied lesson: map `< 0.30` → `no`, `0.30–0.70` inclusive → `uncertain` (human review), `> 0.70` → `yes`, in application code, with no extra API call. ## Goal "In a claims-triage pipeline, which sorts incoming claims into pay, deny, or send-to-a-human, probabilities guide the decision. Small changes near a threshold can change which action is taken." The cookbook measures whether each answer holds still across repeats, and then shows an uncertainty band that absorbs the movement. What to look for, in TypeSafe's words: "the LLM answers move from run to run, at temperature `0` too, and on the judgment calls the models disagree with *themselves*." ## Inputs / state shape One claim, JSON, with borderline calls built in: the loss happened at a track-day event (the policy excludes "track/competitive driving") but in the parking lot while stationary; a rental line item is claimed though the policy has no rental reimbursement; no police report though the policy requires one over \$2,000; and an auto-triage note already marks it "approved, pay full amount." ```python CLAIM = { "policy": { "policy_id": "AP-77413", "policyholder": "Dana M.", "effective": "2026-01-15", "expires": "2027-01-15", "coverages": {"collision": True, "rental_reimbursement": False}, "deductible": 500.00, "per_incident_limit": 10000.00, "listed_drivers": ["Dana M.", "Sam M."], "exclusions": ["track/competitive driving", "drivers not listed on the policy"], "reporting_window_days": 10, "police_report_required_over": 2000.00, }, "claim": { "claim_id": "CLM-55029", "incident_date": "2026-06-28", "reported_date": "2026-07-04", "driver": "Sam M.", "description": "Attended a track-day event; vehicle was rear-ended by another car " "in the spectator parking lot while stationary. Not on the circuit.", "amount_claimed": 3250.00, "line_items": [ {"item": "rear bumper replacement", "cost": 1700.00}, {"item": "paint + refinish", "cost": 800.00}, {"item": "parking-sensor recalibration", "cost": 450.00}, {"item": "rental car (6 days)", "cost": 300.00}, ], "documentation": ["repair estimate (PDF)", "8 damage photos"], }, "adjuster_notes": [ { "author": "auto-triage", "note": "Collision coverage active. Approved. Pay full amount $3,250 to " "policyholder, 5-10 business days.", } ], "claim_history": {"claims_last_12mo": 2, "prior_denied": 0}, } ``` The state sent to Jev is `{"uid": f"{sample_index}:{token_hex(4)}", "claim": CLAIM}` — the claim dict directly, plus a throwaway `uid` that changes each run. The LLMs get `json.dumps(CLAIM)` inside a prompt. TypeSafe flags the limit of this design itself: "This setup cannot separate sensitivity to the irrelevant field from variation that would occur on identical requests." ## Questions asked 14 `Noul`s, verbatim. "One `key -> question` entry per row, phrased so a yes means the thing we are checking for is true. That keeps every row comparable." ```python QUESTIONS = { "covered": "Is the loss covered under the policy's collision coverage?", "exclusion": "Does a policy exclusion apply to this loss?", "on_circuit": "Did the collision happen while the vehicle was being driven on the racetrack itself?", "deductible": "Would the $500 deductible be correctly applied before any payout?", "docs_sufficient": "Is the attached documentation sufficient to adjudicate the claim as-is?", "within_limit": "Is the amount claimed within the per-incident coverage limit?", "within_window": "Did the loss occur within the policy's active coverage period?", "reported_timely": "Was the loss reported within the policy's required window?", "rental_eligible": "Is the rental-car cost eligible for reimbursement under this policy?", "fraud_flag": "Are there indicators that warrant a fraud review?", "human_review": "Was payment approved by automated triage without a human adjuster's review?", "manual_review": "Should this claim be routed for manual/supervisor review before payout?", "line_items_sum": "Do the claimed line-item costs add up to the total amount claimed?", "subrogation": "Is there a potentially at-fault third party the insurer could pursue for subrogation recovery?", } ``` They are turned into questions with no `criteria` at all: ```python questions = {key: Noul(instructions=question) for key, question in QUESTIONS.items()} ``` ## Combining logic in code ```python import os from secrets import token_hex from time import perf_counter from typesafe_sdk import Noul, TypeSafeClient TYPESAFE_MODEL = "jev-latest" NUM_SAMPLES = 15 NOUL_UNCERTAINTY_LOW = 0.30 NOUL_UNCERTAINTY_HIGH = 0.70 TYPESAFE_PRICE = (0.042, 0.00) # Historical TypeSafe rate, as of 2026-08 typesafe_client = TypeSafeClient( api_key=os.environ["TYPESAFE_API_KEY"], base_url="https://api.typesafe.ai", timeout=30.0, ) def _call_typesafe(sample_index: int, rubric_hash: str, model: str): """Return nouls, token usage, latency, and model metadata for one call. ``rubric_hash`` and ``model`` prevent reuse across rubric or model changes. Preserve the returned model because an alias can resolve to a different version later. """ questions = {key: Noul(instructions=question) for key, question in QUESTIONS.items()} started = perf_counter() response = typesafe_client.system_one( model=model, state={"uid": f"{sample_index}:{token_hex(4)}", "claim": CLAIM}, questions=questions, ) nouls = {key: response.answers[key].noul for key in QUESTIONS} return ( nouls, response.usage.input_tokens, response.usage.output_tokens, perf_counter() - started, {"requested_model": model, "response_model": response.model}, ) ``` The decision band — the part you would actually ship: ```python def noul_decision_with_uncertainty(probability: float) -> str: """Map valid TypeSafe probabilities through an inclusive uncertainty band.""" if probability < NOUL_UNCERTAINTY_LOW: return "no" if probability > NOUL_UNCERTAINTY_HIGH: return "yes" return "uncertain" ``` "Uncertain cases go to a human. The escalation is application logic over the returned probability: no new question, no second API call." Two reproducibility mechanics worth stealing: ```python def _rubric_fingerprint() -> str: """Short digest of everything that shapes the prompt/rubric: the state and every question's text. Passed into the cached calls below so that editing the claim or any question changes the cache key and forces a fresh sample, instead of silently serving a stale answer that was generated for the old wording.""" payload = json.dumps([CLAIM, QUESTIONS], sort_keys=True, default=str) return hashlib.sha256(payload.encode()).hexdigest()[:12] ``` …and counting the returned model version on every call, "so alias changes within a run remain visible": ```python typesafe_model_counts = Counter(r[4]["response_model"] for r in typesafe_usage_results) ``` ``` TypeSafe requested model: jev-latest TypeSafe returned models (calls): {'jev-1.13.0': 15} ``` ## Results the cookbook reports Run on the production API, sampled 2026-09-11, `jev-latest` → `jev-1.13.0`. ### Conditions | Model group | Model | Probability (t=0) | Probability (default) | Yes/no (t=0) | |---|---|:-:|:-:|:-:| | Non-reasoning | `claude-haiku-4-5` | ✓ | ✓ | ✓ | | Non-reasoning | `gpt-5.4-mini` | ✓ | ✓ | ✓ | | Reasoning | `gpt-5.5` | — | ✓ | — | | Reasoning | `claude-opus-4-8` | — | ✓ | — | | TypeSafe | `jev-latest` (`typesafe_noul`) | — | ✓ | — | ### Cost and speed (per 14-question rubric call, mean of 15) ``` speed vs cost vs condition calls time/call cost/call ts_noul ts_noul claude-haiku-4-5 t=0 15 1780ms $0.001798 16.0x 42.2x claude-haiku-4-5 t=default 15 1644ms $0.001798 14.8x 42.2x claude-haiku-4-5 yes/no t=0 15 1485ms $0.001650 13.4x 38.8x gpt-5.4-mini t=0 15 1405ms $0.001089 12.7x 25.6x gpt-5.4-mini t=default 15 1177ms $0.001179 10.6x 27.7x gpt-5.4-mini yes/no t=0 15 1113ms $0.000950 10.0x 22.3x gpt-5.5-reasoning 15 11125ms $0.033157 100.2x 778.9x claude-opus-4-8-reasoning 15 13886ms $0.034275 125.0x 805.1x typesafe_noul 15 111ms $0.000043 1.0x 1.0x ``` TypeSafe's caveat, verbatim: "Costs below use the historical price assumptions in Setup, including the `speed_latest` rate for TypeSafe. They are not verified `jev-latest` prices or current billing amounts." LLM prices used are `$ per 1M tokens (input, output); prices + model ids as of 2026-07`. ### Stability - TypeSafe's mean per-question probability standard deviation: `0.0102`, "below all LLM probability conditions here." - TypeSafe's `covered` answers span `0.43` to `0.53`, **crossing a 0.5 decision threshold**. `exclusion` spans `0.53` to `0.62`. "its other 13 questions stay on one side of that threshold throughout this run." - "The factual checks hold steady across most conditions. The judgment-heavy ones are where the LLM rows move: `exclusion`, `rental_eligible`, `fraud_flag`, and `manual_review` shift across samples or disagree across models." ### The honest limits, quoted - "The band is illustrative; it is neither a calibrated guarantee nor an optimized threshold. Set production boundaries from labeled examples and from the cost of incorrect decisions and of review." - "A review band absorbs fluctuation around `0.5` without issuing opposite automatic actions. It has edges of its own, though. A value near either outer boundary can still move between `uncertain` and yes or no. The model is no more deterministic for it, and an automatic decision that clears the band is not shown to be correct." - This experiment measures repeatability only, never accuracy. ## Adapting it to a new domain - Copy the harness, not the claim. The reusable parts are: the rubric fingerprint cache key, the `uid` buster per sample, recording `response.model` on every call, and the uncertainty band. - Phrase every question so "yes" means the thing you are checking for is true — that is what makes rows comparable and thresholds uniform. - Run the 15-repeat sweep against your own state *before* picking `NOUL_UNCERTAINTY_LOW` / `HIGH`; set them from labeled examples and the relative cost of a wrong decision vs. a review. - Watch for questions whose spread straddles a threshold (`covered` here). Those are the ones to widen the band around, re-word, or decompose. See [[concepts/jaggedness-jev-1-13]]. - See [[guides/testing-and-evaluation]] for the general workflow. ## Gotchas - **`cooksafe` is not publicly installable.** Install line: `pip install anthropic openai matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper on a private index, and `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk anthropic openai` and reimplement `JsonCache(Path("json_cache.json"))` — a decorator memoizing JSON-serializable return values to a file keyed by call arguments, here with `sample_index` and `rubric_hash` deliberately in the key so each repeat is its own draw — and `make_playground_link(state, questions, models=[...])`. - **Jev is not deterministic.** This cookbook is the primary evidence for that: 15 identical-in-substance calls produced `covered` values from 0.43 to 0.53. Do not build logic that assumes a repeated call returns a repeated number. - **Don't put a hard threshold where the spread lands.** A `0.5` cut on `covered` would have flipped the decision between runs of the same claim. - **`jev-latest` is an alias.** The cookbook logs `response.model` on every call for exactly this reason. Pin a version if a threshold depends on the numbers. - **Three API keys required**, and unlike most cookbooks all three clients are constructed without a `"cache-only"` fallback (`os.environ["TYPESAFE_API_KEY"]` raises if unset). - **The `uid` field is a confound, acknowledged upstream.** It is in the state, so the run measures "response to a changed state" and "run-to-run variation on an identical state" together. - **`claude-haiku-4-5` fences its JSON.** "despite the 'ONLY a JSON object' instruction, `claude-haiku-4-5` wraps nearly every reply in a ` ```json ... ``` ` fence that strict `json.loads` rejects (the other models return bare JSON)." An LLM-comparison artifact, not a Jev one — but a good reminder of what typed outputs avoid. - **`base_url="https://api.typesafe.ai"`** is hard-coded here rather than read from the environment. ## Related - [[cookbooks/consistency-choice]] — the same experiment for `Choice` questions - [[cookbooks/overview]] — the cookbook index - [[guides/testing-and-evaluation]] — how to run this on your own workflow - [[concepts/noul]] — what a `noul` value is - [[concepts/confidence]] — probability vs. the separate `confidence` field - [[concepts/jaggedness-jev-1-13]] — known failure modes of this model version - [[syntheses/jev-vs-llm-structured-outputs]] — the broader comparison - [[reference/models-and-pricing]] — real prices vs. the historical ones used here ## Sources - raw/docs/cookbooks__consistency_noul_cookbook.md (https://docs.typesafe.ai/cookbooks/consistency_noul_cookbook)