--- title: "Cookbook: Self-consistency — choices" type: cookbook tags: [cookbook, consistency, choice, moderation, uncertainty] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__consistency_choice_cookbook.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "Runs an 8-Choice moderation rubric 15 times across Jev and six LLM conditions; a 0.60 top-probability floor lifts Jev's decision agreement from 90.8% to 99.2% at 74.2% automation." --- # Cookbook: Self-consistency — choices > **TL;DR** A repeatability experiment. One borderline moderation post, an 8-question `Choice` rubric, 15 repeats per condition, Jev (`jev-latest` → `jev-1.13.0`) vs. six LLM conditions. Jev's raw label agreement is 90.8% and its mean probability std dev `0.0098`; adding `uncertain` when the top probability is below `0.60` raises decision agreement to **99.2%** while still acting automatically on **74.2%** of answers. Per call: 114ms, \$0.000046. ## Goal "In a moderation pipeline that label is the routing decision: remove or leave up, escalate or auto-resolve, send to the threat, spam, or general queue. When the label wobbles from one run to the next, the same post routes to different places for no good reason." What to look for, per TypeSafe: "picked labels can flip inside a single condition, including TypeSafe, and conditions disagree with each other." ## Inputs / state shape One post, "built to sit on the fence": heated insults aimed partly at a person and partly at the community, an off-platform `discord.gg` invite, one prior strike, four user reports, and threat-like wording that is "never cleanly phrased." ```python POST = { "post_id": "P-88213", "author": { "user_id": "u/4471", "account_age_days": 38, "prior_strikes": 1, "followers": 210, }, "context": { "surface": "public reply", "in_reply_to": "another user defending a game patch", "community": "r/gamedebates", }, "content": { "text": ( "Are you seriously this dense? Anyone who defends that patch is a complete clown and " "should be embarrassed to even post here. People like you are what's ruining this " "community and honestly you need to be dealt with. Come say it to my face, invite's " "right here. Keep it up and I'll end your whole channel." ), "has_link": True, "link_domain": "discord.gg", "language": "en", }, "reports": {"user_reports": 4, "report_reasons": ["harassment", "spam", "threat"]}, } ``` Jev's state is `{"uid": f"{rubric_hash}:{sample_index}:{token_hex(4)}", "post": POST}`. The LLMs get `json.dumps(POST)` in a prompt. Same `uid` confound as the noul cookbook, acknowledged the same way. ## Questions asked Eight `Choice` questions, each a `(instructions, {label: description})` pair, verbatim. "The labels within a question are mutually exclusive (exactly one applies), and each carries a short description." ```python QUESTIONS = { "category": ( "What is the single most applicable content-policy category for this post?", { "None": "No policy violation of any kind.", "Harass": "Insults or demeans a person, with no threat of harm and no protected-class attack.", "Hate": "Attacks a person or group over a protected characteristic (race, religion, gender, ...).", "Violence": "Makes a credible threat of harm or incites violence against someone.", "Spam": "Unsolicited promotion or link spam, with no personal attack.", "Sexual": "Sexual or adult content.", }, ), "primary_risk": ( "What is the primary moderation risk that should drive triage for this post?", { "Harassment": "Personal attack or targeted abuse is the main risk.", "Violence": "A threat of harm or intimidation is the main risk.", "LinkAbuse": "External-link or off-platform coordination risk is the main risk.", "AccountHistory": "Prior account history or repeat behavior is the main risk.", "LowRisk": "No meaningful moderation risk is present.", }, ), "target": ( "Who or what is the content primarily directed at?", { "None": "Not directed at anyone in particular.", "Person": "Directed at one specific individual.", "Group": "Directed at a protected group or class.", "Platform": "Directed at the community or platform itself, not a person.", }, ), "action": ( "What enforcement action should be taken on this post?", { "Allow": "Leave the post up with no action.", "Warn": "Leave the post up but attach a warning label.", "Remove": "Remove the post, but do not penalize the account.", "Strike": "Remove the post and add a strike to the account.", "Escalate": "Take no automated action; hold for a human decision.", }, ), "queue": ( "Which single moderation queue should own this post?", { "Auto": "Auto-resolve; no human queue needed.", "General": "General moderation queue.", "Threat": "Threat / violence response queue.", "Spam": "Spam and platform-abuse queue.", "TSLead": "Trust-and-safety lead / senior queue.", }, ), "link_handling": ( "How should any external link or off-platform invite in the post be handled?", { "Allow": "Leave the link in place.", "RmLink": "Strip or disable the link but keep the post.", "Brigade": "Treat the link as coordinated brigading and action it as abuse.", "Escalate": "Send the link to a specialist to assess before acting.", }, ), "review_path": ( "Who should make the final call on this post?", { "Auto": "Automated action; no human review.", "Human": "A frontline human moderator makes the call.", "Senior": "A senior or specialist reviewer is required.", "Legal": "Route to legal or law-enforcement escalation.", }, ), "severity": ( "What is the overall severity of this post?", { "None": "No violation.", "Low": "Rude or dismissive, but essentially harmless.", "Medium": "Personal harassment with no clearly credible threat.", "High": "Harassment together with a threat that could be read as credible.", }, ), } ``` Built into SDK questions with: ```python questions = { key: Choice(instructions=instructions, criteria=choices) for key, (instructions, choices) in QUESTIONS.items() } ``` ## Combining logic in code ```python import os from collections import Counter from secrets import token_hex from time import perf_counter import numpy as np from typesafe_sdk import Choice, TypeSafeClient TYPESAFE_MODEL = "jev-latest" NUM_SAMPLES = 15 MIN_CHOICE_PROBABILITY = 0.60 # illustrative automatic-action threshold TYPESAFE_PRICE = (0.042, 0.00) # Historical TypeSafe rate, as of 2026-08 typesafe_client = TypeSafeClient( api_key=os.environ["TYPESAFE_API_KEY"], base_url="https://api.typesafe.ai", timeout=30.0, ) def _call_typesafe(sample_index: int, rubric_hash: str, model: str): questions = { key: Choice(instructions=instructions, criteria=choices) for key, (instructions, choices) in QUESTIONS.items() } started = perf_counter() response = typesafe_client.system_one( model=model, state={"uid": f"{rubric_hash}:{sample_index}:{token_hex(4)}", "post": POST}, questions=questions, ) distributions = {} for key, (_instructions, choices) in QUESTIONS.items(): probabilities = dict(response.answers[key].probabilities) distributions[key] = [probabilities.get(label, float("nan")) for label in choices] return ( distributions, response.usage.input_tokens, response.usage.output_tokens, perf_counter() - started, {"requested_model": model, "response_model": response.model}, ) ``` The decision rule — the shippable part: ```python def argmax_label(values: list, labels: list[str]) -> str | None: """The label with the most probability mass, or ``None`` if any value is missing or non-numeric -- a partially parsed distribution never yields a confident-looking pick.""" numeric = [_numeric_value(value) for value in values] if any(value is None for value in numeric): return None return labels[int(np.argmax(numeric))] def choice_decision_with_uncertainty(values: list, labels: list[str]) -> str | None: """Abstain below the action threshold; retain invalid results as parse failures.""" label = argmax_label(values, labels) if label is None: return None probabilities = [float(value) for value in values] if any(value < 0 or value > 1 for value in probabilities): return None return label if max(probabilities) >= MIN_CHOICE_PROBABILITY else "uncertain" ``` An important implementation note from the text: "This uses the returned probabilities, not the API's separate `confidence` field, and adds no model calls." At exactly `0.60`, the top label is selected. ## Results the cookbook reports Production API, sampled 2026-09-11. All 15 Jev calls returned `jev-1.13.0`. ### Cost and speed (per 8-question rubric call, mean of 15; LLMs in a 16-way pool) ``` speed vs cost vs condition calls time/call cost/call ts_choice ts_choice claude-haiku-4-5 t=0 15 3853ms $0.003498 33.8x 76.1x claude-haiku-4-5 t=default 15 3860ms $0.003494 33.8x 76.0x claude-haiku-4-5 single-pick t=0 15 992ms $0.001527 8.7x 33.2x gpt-5.4-mini t=0 15 2293ms $0.002299 20.1x 50.0x gpt-5.4-mini t=default 15 1986ms $0.002164 17.4x 47.1x gpt-5.4-mini single-pick t=0 15 826ms $0.000936 7.2x 20.3x gpt-5.5-reasoning 15 12978ms $0.041255 113.7x 897.4x claude-opus-4-8-reasoning 15 10376ms $0.028375 90.9x 617.2x typesafe_choice 15 114ms $0.000046 1.0x 1.0x ``` Same caveat as the noul cookbook: these use "historical price assumptions… not verified `jev-latest` prices or current billing amounts." TypeSafe also notes its own calls were drawn sequentially "after the LLM pool has closed, so each call's latency is a clean round trip rather than one measured under the 16-way LLM thread contention." ### Probability standard deviation Method: per condition, std dev of each label's probability across the 15 repeats, averaged over all labels and questions. ``` condition mean prob std max prob std parse fail x TypeSafe claude-haiku-4-5 t=0 0.0012 0.0221 0% 0.12x claude-haiku-4-5 t=default 0.0516 0.3150 1% 5.29x gpt-5.4-mini t=0 0.0312 0.0905 0% 3.20x gpt-5.4-mini t=default 0.0543 0.2303 0% 5.56x gpt-5.5-reasoning 0.0305 0.1047 0% 3.12x claude-opus-4-8-reasoning 0.0245 0.0693 0% 2.52x typesafe_choice 0.0098 0.0515 0% 1.00x ``` TypeSafe reports this honestly: "Haiku at temperature 0 has a lower mean std dev of `0.0012`. The other five LLM probability conditions range from `0.0245` to `0.0543`, about `2.5x` to `5.6x` the TypeSafe mean. Small changes can still switch the top label when two labels are close." ### Agreement, with and without the uncertain band ``` condition raw agree policy agree uncertain automatic conflicts claude-haiku-4-5 t=0 100.0% 100.0% 0.0% 100.0% 0 claude-haiku-4-5 t=default 87.5% 86.7% 0.8% 98.3% 2 gpt-5.4-mini t=0 99.2% 87.5% 12.5% 87.5% 0 gpt-5.4-mini t=default 90.8% 84.2% 22.5% 77.5% 2 gpt-5.5-reasoning 90.0% 93.3% 30.8% 69.2% 1 claude-opus-4-8-reasoning 92.5% 94.2% 33.3% 66.7% 0 typesafe_choice 90.8% 99.2% 25.8% 74.2% 0 ``` Column definitions from the cookbook: "`policy agree` counts `uncertain` as a decision; parse failures count against agreement. `automatic` is the share of all answers that select a label. `conflicts` counts questions with more than one concrete label across the repeats, ignoring abstentions." Where Jev moved: "Before abstention, TypeSafe changes its top label on `primary_risk` (Harassment 11 times, Violence 4 times) and `link_handling` (RmLink 8 times, Brigade 7 times). Both rows now show `uncertain` throughout because their top probabilities are below `0.60`." Also: "`category` alternated between Violence and `uncertain`, crossing the action threshold on some repeats and not others. No question produced two different concrete TypeSafe labels." Steady questions: "`target` reads Person and `severity` reads High across the board." Contested ones: `category`, `primary_risk`, `action`, `review_path`, `link_handling`. The refusal to over-claim, quoted: "None of this shows accuracy or superiority: Haiku at temperature 0 had 100% agreement here, with no abstentions." And on the chart itself: "Haiku t=0: 100% repeatability does not imply correctness. This experiment does not measure accuracy." ## Adapting it to a new domain - Copy the harness: rubric fingerprint as cache key, per-sample `uid`, logging `response.model`, and `choice_decision_with_uncertainty`. - Make every question's label set mutually exclusive and give every label a one-line description; that is what makes the abstention rule meaningful. - Sweep `MIN_CHOICE_PROBABILITY` against your own labeled examples. The cookbook is explicit: "The threshold is an illustrative application policy, not a calibrated guarantee or a threshold chosen to maximize this run's agreement." - Read the abstention rate as a cost: 0.60 bought 99.2% agreement at the price of routing 25.8% of answers to a human. A lower floor automates more and agrees less. - Questions that abstain on *every* repeat (`primary_risk`, `link_handling` here) are telling you the rubric is under-specified for this input, not that the model failed. Re-word or decompose them — see [[guides/writing-instructions-and-criteria]]. ## Gotchas - **`cooksafe` is not publicly installable.** Install line: `pip install anthropic openai matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper on a private index, and `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk anthropic openai` and reimplement `JsonCache(Path("json_cache.json"))` (a decorator memoizing JSON-serializable returns to a file keyed by call arguments, with `sample_index` + `rubric_hash` in the key so each repeat is an independent draw) and `make_playground_link(state, questions, models=[...])`. - **Jev's labels can flip.** 2 of 8 questions changed top label within 15 repeats of the same post. Do not treat a `Choice` as deterministic. - **The abstention band has its own edges.** "a probability near `0.60` can still move between a concrete label and `uncertain`." - **Use `probabilities`, not `confidence`, for this rule** — the cookbook says so explicitly, and they are different fields. See [[concepts/confidence]]. - **`probabilities.get(label, float("nan"))`** — the code does not assume the response contains every label. Preserve that defensiveness. - **Repeatability is not accuracy.** Stated three separate times in the source, including inside the exported chart. - **`jev-latest` is an alias**; `response.model` is logged per call so an alias move mid-run stays visible. ## Related - [[cookbooks/consistency-noul]] — the same experiment for `Noul` questions - [[cookbooks/overview]] — the cookbook index - [[guides/testing-and-evaluation]] — running this on your own workflow - [[concepts/choice]] — `choice`, `probabilities`, `confidence` - [[concepts/confidence]] — why the two are not interchangeable - [[concepts/jaggedness-jev-1-13]] — known failure modes of this model version - [[patterns/intent-routing]] — routing on a Choice label in production - [[syntheses/jev-vs-llm-structured-outputs]] — the broader comparison ## Sources - raw/docs/cookbooks__consistency_choice_cookbook.md (https://docs.typesafe.ai/cookbooks/consistency_choice_cookbook)