$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/syntheses

Jev vs LLM JSON mode / structured outputs

[ synthesis ][ updated 2026-09-17 ][ confidence medium ][ jev-1.13.0 ][ python sdk 0.6.0 ]#comparison · structured-outputs · json-mode · llm · decision-guide

TL;DR JSON mode constrains the text an LLM writes; Jev never writes text, so the answer space is the schema and every answer arrives with a probability distribution over it. Pick Jev for narrow, high-volume judgments where code consumes the answer and calibrated uncertainty matters; pick an LLM for anything that must be generated, explained, multi-step, or multimodal. In most real systems the answer is both — Jev as the router, guardrail, or judge around an LLM that does the writing.

The question

"How is this different from JSON mode or structured outputs?" is question 3 on TypeSafe's homepage FAQ, and its answer is not in the captured page (raw/site/typesafe-ai-home.txt — the accordions are collapsed). This page answers it from the documentation, the launch post, TypeSafe's own LLM adapter, and the cookbooks that pit the two against each other. Everything attributed to TypeSafe below is TypeSafe's claim about its own product; no independent reproduction of any performance number exists in the sources (Press and third-party coverage).

The mechanism difference

An LLM with JSON mode or structured outputs is still an autoregressive text model. The provider constrains decoding so the emitted string parses against your schema: OpenAI's Responses API takes {"type": "json_schema", "name": ..., "schema": ..., "strict": true} (or {"type": "json_object"} for plain JSON mode), and Anthropic's native API takes output_config: {"format": {"type": "json_schema", "schema": ...}} — both documented in TypeSafe's own adapter, which drives them (system-one-adapter: LLM-backed drop-in for TypeSafeClient). The model still writes the answer, token by token, and any probability you want has to be written out as a number inside that JSON, by the same process that wrote the answer.

A System One model removes the writing step. Possible outputs are fixed in advance by your criteria, a parallel sampler emits all outputs in a single query rather than one token at a time, and what you get back is a distribution over your options — the probabilities are the model's own, not a number it chose to type. This is the whole architectural claim in the launch post (Blog: Introducing System One Models & Jev (2026-09-15)), and it is why TypeSafe says Jev "never makes type errors": schema conformance is structural rather than best-effort.

The corollary matters as much as the claim: typed output guarantees the interface, not the truth. Jev can return a wrong but perfectly valid answer, and TypeSafe's own agent skill says so in those words (The typesafe-ai agent skill and Claude Code plugin, Jev 1.13 jaggedness: known failure modes).

Comparison

Rows marked TypeSafe's characterization are TypeSafe describing a competitor category; treat them as claims. Rows marked not in sources are gaps this wiki will not fill by guessing.

Dimension Jev (System One) LLM with JSON mode / structured outputs Source
Output type Typed values drawn from criteria you defined in advance; never a string the model composed A string the decoder constrained to parse against a schema; your code then validates it Primitives: Choice, Score, Noul, system-one-adapter: LLM-backed drop-in for TypeSafeClient
Probabilities probabilities over every Choice option and every Score level, summing to ~1, returned on every answer Only if you ask the model to write numbers into the JSON; TypeSafe's adapter needs a normalize_probabilities option because LLM distributions arrive invalid Confidence vs probability, system-one-adapter: LLM-backed drop-in for TypeSafeClient
Confidence A 0–1 statistic derived from the distribution, on Choice and Score answers (Noul has none) No native field; TypeSafe's characterization: "models tend to be overconfident and inconsistent even when prompted for a confidence estimate" Confidence vs probability, Blog: Introducing System One Models & Jev (2026-09-15)
Calibration The training objective: RLCD optimises probabilities against outcomes. Calibration is a group property, never a per-answer guarantee TypeSafe's characterization: RLHF optimises human preference and RLVR verifiable rewards, neither of which targets calibration AI primer: why calibrated decision models
Latency Claimed 70–500 ms end to end; TypeSafe's own cookbook runs report 111 ms (14 Nouls) and 114 ms (8 Choices), measured from West Coast laptops TypeSafe's characterization: 3–329 s end to end for frontier models. In the same cookbook, the LLM conditions ran 826 ms – 12,978 ms per equivalent call Models, aliases, pricing, rate limits, context, Cookbook: Self-consistency — choices
Price $0.042 / Mtok input, output tokens free TypeSafe's characterization: $0.20–$10 / MTok input with output ~5x input. The same cookbook's LLM conditions cost $0.000936–$0.041255 per call against Jev's $0.000046 Models, aliases, pricing, rate limits, context, Cookbook: Self-consistency — choices
Context 64k tokens per request; 32k for state plus the longest question Not in sources — no source states any LLM's context window Models, aliases, pricing, rate limits, context
Modality Text only: string, JSON object, or array of text values. Pre-process anything else Not characterised in the sources beyond "unstructured data … with an emphasis on sequential messages"; do not read a multimodality comparison into this wiki State: what you send Jev, Blog: Introducing System One Models & Jev (2026-09-15)
Hallucination / type errors Structurally impossible to return a value outside your schema. TypeSafe's plotted 0% "is not empirical. Schema matching is guaranteed" TypeSafe plots LLM type-error rates from OpenRouter data and flags the bias; third-party write-ups record most models between 0.58% and 13.2%, with a single 45.5% outlier Workflow evals: how TypeSafe measures Jev, Press and third-party coverage
Determinism / repeatability No determinism claim, no temperature or seed parameter. Measured: mean probability std dev 0.0098 (Choice) and 0.0102 (Noul) over 15 repeats; raw top-label agreement 90.8% → 99.2% with an uncertain band Measured in the same harness: "the LLM answers move from run to run, at temperature 0 too". Mean std devs 0.0245–0.0543 for five conditions — but Haiku 4.5 at t=0 scored 0.0012 and 100% agreement, better than Jev Cookbook: Self-consistency — choices, Cookbook: Self-consistency — nouls
Generation None. "Not trained to generate text"; forcing it by chaining choices "will not work well and will be very slow" The thing it is for Jev 1.13 jaggedness: known failure modes
Reasoning / indirection Degrades with each hop of indirection; no chain of thought, no self-chosen next action The thing RLVR reasoning models are for; the sources use them as the expensive escalation tier Jev 1.13 jaggedness: known failure modes, Cookbook: SDE cascade
Failure mode when the right answer is missing Probability still lands somewhere — Choice options always sum to 1 — so you must supply an explicit other / none route Can invent a schema-valid but fabricated value; schema validation "catches structural errors and never semantic ones" Choice questions, Cookbook: SDE cascade
Tooling One endpoint, an OpenAPI 0.2.0 document, Python and JS SDKs at 0.6.0, an agent skill, a console playground, and an LLM adapter for comparisons Provider-native; the adapter exists precisely so the same question code runs on either side HTTP API: POST /v1/systemone and GET /v1/models, system-one-adapter: LLM-backed drop-in for TypeSafeClient

When to pick which

Situation Pick Why
Classify, route, rank, score, detect, or verify, at volume Jev Narrow answer space, code consumes the result, price and latency dominate (Use-case map by industry)
The decision needs a number you can threshold and audit Jev Calibrated probabilities and confidence are first-class, not prose (Confidence vs probability)
The output is prose, code, a summary, or an explanation LLM Jev cannot generate (Jev 1.13 jaggedness: known failure modes)
The task needs several reasoning hops, or the model must choose its own next action LLM Explicitly outside the System One contract (System One Models)
Input is an image, audio, or video Neither directly — pre-process to text, then Jev Jev is text-only (State: what you send Jev)
Arithmetic, counting, date ordering Code Jaggedness modes 2 and 3; extract with Jev, compute in code (Cookbook: Date extraction)
Answer space is large but enumerable (up to 255 options) Jev A Choice over the enumerated options cannot invent a value (Cookbook: Pre-parsed value extraction)
Answer space is open-ended text LLM to propose, Jev to select "Select instead of generate" (The typesafe-ai agent skill and Claude Code plugin)
You need an explanation of why for a human reviewer LLM alongside Jev Jev returns no rationale; the distribution is the only trace you get

How to combine them

The cookbooks treat this as the normal case, not a compromise. Three shapes:

Jev in front — router. One cheap call returns intent (Choice) plus a complexity Score; code sends the easy cases to a database lookup, the hard ones to a specialist LLM, and low-confidence ones to a human. "The expensive resources only get invoked for the requests that actually need them" — Intent routing, and Cookbook: Function calling for the version that also fills the handler's typed arguments.

Jev around — guardrail. One system_one call per message screens every input to and output from an LLM app: four hazard Nouls plus a severity Score, thresholded in your code into pass, review, block, or support. The same cached assessment routes differently under a strict or permissive policy because the policy lives in your code, not the prompt — Cookbook: Guardrails for LLMs. The retrieval-side twin drops prompt injections and flags contradictions before passages reach the generator — Cookbook: Classifying RAG passages.

Jev after — judge. A cheap LLM extracts, a Jev Noul battery scores P(wrong) per field with questions framed so that true means something is broken, and only records where a flag exceeds 0.7 get re-extracted by an expensive reasoning model. The point is stated bluntly: schema validation catches structural errors and never semantic ones; the verifier is what catches a schema-valid fabrication — Cookbook: SDE cascade. Cookbook: Double-checking citations is the same move for citations, with a string match first and one Choice per surviving quote.

A fourth, quieter shape: Jev as feature extractor feeding a classical model — Score and Noul answers become numeric columns for a supervised regressor (Cookbook: Autoresearch feature discovery).

How to benchmark the two fairly

  1. Hold the harness fixed. TypeSafe's own methodology is to assume the workflow code is correct and change only the model behind it (Workflow evals: how TypeSafe measures Jev). Its most portable finding is not about Jev at all: every model tested was more accurate, cheaper and faster expressed as a structured workflow than as one prompt.
  2. Run both from the same question definitions. system-one-adapter answers typesafe_sdk questions with an OpenAI or Anthropic model and returns the same NoulAnswer / ChoiceAnswer / ScoreAnswer objects, so nothing but the client changes (system-one-adapter: LLM-backed drop-in for TypeSafeClient).
  3. Decide the LLM's answer mode deliberately, and report it. llm_answer_mode="probabilities" versus "discrete", and structured_outputs=True versus prompted JSON, change both the cost and what is comparable. TypeSafe concedes its own wrapper "tends to be slower and more expensive than giving decisions without probabilities" — that overhead is part of why its multipliers are large.
  4. Count everything the LLM side spends. The adapter's usage exposes input_tokens_total / output_tokens_total across attempts, n_retries, n_retries_malformed_structure, and latency; a comparison that ignores corrective retries flatters the LLM.
  5. Repeat, and report variance. The consistency cookbooks run 15 repeats per condition and report mean and max probability standard deviation plus parse-failure rate. One run of each side is not an evaluation (Cookbook: Self-consistency — choices, Testing and evaluating a Jev workflow).
  6. Build reference labels honestly, and say what they are. TypeSafe averages two frontier models at high thinking — which means its own score is bounded by them by construction, and that "accuracy" means agreement, not correctness. If you have real outcome labels, use those instead.
  7. Check the contract before you publish. MCA §2.3(f) prohibits publishing "benchmarks or performance information about the Services." Running private evals is exactly what TypeSafe's own anti-benchmaxxing post recommends; publishing them is a separate question for your legal team (Legal: MCA, DPA, privacy, data retention, Blog: Lies, Damned Lies, and Benchmarks).

Caveats on TypeSafe's own claims

Related

Sources