$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Self-consistency — nouls

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · consistency · noul · evaluation · uncertainty

TL;DR A repeatability experiment, not a recipe. One auto-insurance claim, a 14-question Noul rubric, 15 repeats per condition, across Jev (jev-latest, resolved to jev-1.13.0 in all 15 calls) and six LLM conditions. TypeSafe reports a mean per-question probability standard deviation of 0.0102, "below all LLM probability conditions here," at 111ms and $0.000043 per 14-question call. The applied lesson: map < 0.30 → no, 0.30–0.70 inclusive → uncertain (human review), > 0.70 → yes, in application code, with no extra API call.

Goal

"In a claims-triage pipeline, which sorts incoming claims into pay, deny, or send-to-a-human, probabilities guide the decision. Small changes near a threshold can change which action is taken." The cookbook measures whether each answer holds still across repeats, and then shows an uncertainty band that absorbs the movement.

What to look for, in TypeSafe's words: "the LLM answers move from run to run, at temperature 0 too, and on the judgment calls the models disagree with themselves."

Inputs / state shape

One claim, JSON, with borderline calls built in: the loss happened at a track-day event (the policy excludes "track/competitive driving") but in the parking lot while stationary; a rental line item is claimed though the policy has no rental reimbursement; no police report though the policy requires one over $2,000; and an auto-triage note already marks it "approved, pay full amount."

CLAIM = {
    "policy": {
        "policy_id": "AP-77413",
        "policyholder": "Dana M.",
        "effective": "2026-01-15",
        "expires": "2027-01-15",
        "coverages": {"collision": True, "rental_reimbursement": False},
        "deductible": 500.00,
        "per_incident_limit": 10000.00,
        "listed_drivers": ["Dana M.", "Sam M."],
        "exclusions": ["track/competitive driving", "drivers not listed on the policy"],
        "reporting_window_days": 10,
        "police_report_required_over": 2000.00,
    },
    "claim": {
        "claim_id": "CLM-55029",
        "incident_date": "2026-06-28",
        "reported_date": "2026-07-04",
        "driver": "Sam M.",
        "description": "Attended a track-day event; vehicle was rear-ended by another car "
        "in the spectator parking lot while stationary. Not on the circuit.",
        "amount_claimed": 3250.00,
        "line_items": [
            {"item": "rear bumper replacement", "cost": 1700.00},
            {"item": "paint + refinish", "cost": 800.00},
            {"item": "parking-sensor recalibration", "cost": 450.00},
            {"item": "rental car (6 days)", "cost": 300.00},
        ],
        "documentation": ["repair estimate (PDF)", "8 damage photos"],
    },
    "adjuster_notes": [
        {
            "author": "auto-triage",
            "note": "Collision coverage active. Approved. Pay full amount $3,250 to "
            "policyholder, 5-10 business days.",
        }
    ],
    "claim_history": {"claims_last_12mo": 2, "prior_denied": 0},
}

The state sent to Jev is {"uid": f"{sample_index}:{token_hex(4)}", "claim": CLAIM} — the claim dict directly, plus a throwaway uid that changes each run. The LLMs get json.dumps(CLAIM) inside a prompt. TypeSafe flags the limit of this design itself: "This setup cannot separate sensitivity to the irrelevant field from variation that would occur on identical requests."

Questions asked

14 Nouls, verbatim. "One key -> question entry per row, phrased so a yes means the thing we are checking for is true. That keeps every row comparable."

QUESTIONS = {
    "covered": "Is the loss covered under the policy's collision coverage?",
    "exclusion": "Does a policy exclusion apply to this loss?",
    "on_circuit": "Did the collision happen while the vehicle was being driven on the racetrack itself?",
    "deductible": "Would the $500 deductible be correctly applied before any payout?",
    "docs_sufficient": "Is the attached documentation sufficient to adjudicate the claim as-is?",
    "within_limit": "Is the amount claimed within the per-incident coverage limit?",
    "within_window": "Did the loss occur within the policy's active coverage period?",
    "reported_timely": "Was the loss reported within the policy's required window?",
    "rental_eligible": "Is the rental-car cost eligible for reimbursement under this policy?",
    "fraud_flag": "Are there indicators that warrant a fraud review?",
    "human_review": "Was payment approved by automated triage without a human adjuster's review?",
    "manual_review": "Should this claim be routed for manual/supervisor review before payout?",
    "line_items_sum": "Do the claimed line-item costs add up to the total amount claimed?",
    "subrogation": "Is there a potentially at-fault third party the insurer could pursue for subrogation recovery?",
}

They are turned into questions with no criteria at all:

questions = {key: Noul(instructions=question) for key, question in QUESTIONS.items()}

Combining logic in code

import os
from secrets import token_hex
from time import perf_counter
from typesafe_sdk import Noul, TypeSafeClient

TYPESAFE_MODEL = "jev-latest"
NUM_SAMPLES = 15
NOUL_UNCERTAINTY_LOW = 0.30
NOUL_UNCERTAINTY_HIGH = 0.70
TYPESAFE_PRICE = (0.042, 0.00)  # Historical TypeSafe rate, as of 2026-08

typesafe_client = TypeSafeClient(
    api_key=os.environ["TYPESAFE_API_KEY"],
    base_url="https://api.typesafe.ai",
    timeout=30.0,
)


def _call_typesafe(sample_index: int, rubric_hash: str, model: str):
    """Return nouls, token usage, latency, and model metadata for one call.

    ``rubric_hash`` and ``model`` prevent reuse across rubric or model changes.
    Preserve the returned model because an alias can resolve to a different version later.
    """
    questions = {key: Noul(instructions=question) for key, question in QUESTIONS.items()}
    started = perf_counter()
    response = typesafe_client.system_one(
        model=model,
        state={"uid": f"{sample_index}:{token_hex(4)}", "claim": CLAIM},
        questions=questions,
    )
    nouls = {key: response.answers[key].noul for key in QUESTIONS}
    return (
        nouls,
        response.usage.input_tokens,
        response.usage.output_tokens,
        perf_counter() - started,
        {"requested_model": model, "response_model": response.model},
    )

The decision band — the part you would actually ship:

def noul_decision_with_uncertainty(probability: float) -> str:
    """Map valid TypeSafe probabilities through an inclusive uncertainty band."""
    if probability < NOUL_UNCERTAINTY_LOW:
        return "no"
    if probability > NOUL_UNCERTAINTY_HIGH:
        return "yes"
    return "uncertain"

"Uncertain cases go to a human. The escalation is application logic over the returned probability: no new question, no second API call."

Two reproducibility mechanics worth stealing:

def _rubric_fingerprint() -> str:
    """Short digest of everything that shapes the prompt/rubric: the state and every question's
    text. Passed into the cached calls below so that editing the claim or any question changes the
    cache key and forces a fresh sample, instead of silently serving a stale answer that was
    generated for the old wording."""
    payload = json.dumps([CLAIM, QUESTIONS], sort_keys=True, default=str)
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

…and counting the returned model version on every call, "so alias changes within a run remain visible":

typesafe_model_counts = Counter(r[4]["response_model"] for r in typesafe_usage_results)

TypeSafe requested model: jev-latest
TypeSafe returned models (calls): {'jev-1.13.0': 15}

Results the cookbook reports

Run on the production API, sampled 2026-09-11, jev-latest → jev-1.13.0.

Conditions

Model group Model Probability (t=0) Probability (default) Yes/no (t=0)
Non-reasoning claude-haiku-4-5 ✓ ✓ ✓
Non-reasoning gpt-5.4-mini ✓ ✓ ✓
Reasoning gpt-5.5 — ✓ —
Reasoning claude-opus-4-8 — ✓ —
TypeSafe jev-latest (typesafe_noul) — ✓ —

Cost and speed (per 14-question rubric call, mean of 15)

                                                               speed vs    cost vs
condition                      calls  time/call    cost/call    ts_noul    ts_noul
claude-haiku-4-5 t=0              15     1780ms    $0.001798      16.0x      42.2x
claude-haiku-4-5 t=default        15     1644ms    $0.001798      14.8x      42.2x
claude-haiku-4-5 yes/no t=0       15     1485ms    $0.001650      13.4x      38.8x
gpt-5.4-mini t=0                  15     1405ms    $0.001089      12.7x      25.6x
gpt-5.4-mini t=default            15     1177ms    $0.001179      10.6x      27.7x
gpt-5.4-mini yes/no t=0           15     1113ms    $0.000950      10.0x      22.3x
gpt-5.5-reasoning                 15    11125ms    $0.033157     100.2x     778.9x
claude-opus-4-8-reasoning         15    13886ms    $0.034275     125.0x     805.1x
typesafe_noul                     15      111ms    $0.000043       1.0x       1.0x

TypeSafe's caveat, verbatim: "Costs below use the historical price assumptions in Setup, including the speed_latest rate for TypeSafe. They are not verified jev-latest prices or current billing amounts." LLM prices used are $ per 1M tokens (input, output); prices + model ids as of 2026-07.

Stability

The honest limits, quoted

Adapting it to a new domain

Gotchas

Related

Sources