---
title: "Jev vs LLM JSON mode / structured outputs"
type: synthesis
tags: [comparison, structured-outputs, json-mode, llm, decision-guide]
created: 2026-09-17
updated: 2026-09-17
confidence: medium
sources:
  - wiki/concepts/system-one.md
  - wiki/concepts/primitives.md
  - wiki/concepts/confidence.md
  - wiki/concepts/machine-learning-primer.md
  - wiki/concepts/jaggedness-jev-1-13.md
  - wiki/concepts/workflow-evals.md
  - wiki/reference/http-api.md
  - wiki/reference/models-and-pricing.md
  - wiki/reference/system-one-adapter.md
  - wiki/reference/legal-and-data.md
  - wiki/cookbooks/consistency-choice.md
  - wiki/cookbooks/consistency-noul.md
  - wiki/cookbooks/llm-guardrails.md
  - wiki/cookbooks/sde-cascade.md
  - wiki/cookbooks/citation-check.md
  - wiki/cookbooks/classifying-rag-passages.md
  - wiki/patterns/intent-routing.md
  - wiki/entities/blog-introducing-system-one.md
  - wiki/entities/press-and-coverage.md
  - raw/site/blog-introducing-system-one.txt
  - raw/docs/cookbooks__consistency_choice_cookbook.md
  - raw/docs/cookbooks__consistency_noul_cookbook.md
  - raw/github/system-one-adapter-python/README.md
  - raw/site/typesafe-ai-legal_mca.txt
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "What Jev gives you that an LLM in JSON mode does not (and vice versa), a row-by-row comparison, when to pick each, how to combine them, and how to benchmark the pair honestly."
---

# Jev vs LLM JSON mode / structured outputs

> **TL;DR** JSON mode constrains the *text* an LLM writes; Jev never writes text, so the answer space **is** the schema and every answer arrives with a probability distribution over it. Pick Jev for narrow, high-volume judgments where code consumes the answer and calibrated uncertainty matters; pick an LLM for anything that must be generated, explained, multi-step, or multimodal. In most real systems the answer is both — Jev as the router, guardrail, or judge around an LLM that does the writing.

## The question

"How is this different from JSON mode or structured outputs?" is question 3 on TypeSafe's homepage FAQ, and its answer is not in the captured page (`raw/site/typesafe-ai-home.txt` — the accordions are collapsed). This page answers it from the documentation, the launch post, TypeSafe's own LLM adapter, and the cookbooks that pit the two against each other. Everything attributed to TypeSafe below is TypeSafe's claim about its own product; no independent reproduction of any performance number exists in the sources ([[entities/press-and-coverage]]).

## The mechanism difference

An LLM with JSON mode or structured outputs is still an autoregressive text model. The provider constrains decoding so the emitted string parses against your schema: OpenAI's Responses API takes `{"type": "json_schema", "name": ..., "schema": ..., "strict": true}` (or `{"type": "json_object"}` for plain JSON mode), and Anthropic's native API takes `output_config: {"format": {"type": "json_schema", "schema": ...}}` — both documented in TypeSafe's own adapter, which drives them ([[reference/system-one-adapter]]). The model still *writes* the answer, token by token, and any probability you want has to be written out as a number inside that JSON, by the same process that wrote the answer.

A [[concepts/system-one|System One model]] removes the writing step. Possible outputs are fixed in advance by your `criteria`, a parallel sampler emits all outputs in a single query rather than one token at a time, and what you get back is a distribution over your options — the probabilities are the model's own, not a number it chose to type. This is the whole architectural claim in the launch post ([[entities/blog-introducing-system-one]]), and it is why TypeSafe says Jev "never makes type errors": schema conformance is structural rather than best-effort.

The corollary matters as much as the claim: **typed output guarantees the interface, not the truth.** Jev can return a wrong but perfectly valid answer, and TypeSafe's own agent skill says so in those words ([[reference/agent-skill]], [[concepts/jaggedness-jev-1-13]]).

## Comparison

Rows marked *TypeSafe's characterization* are TypeSafe describing a competitor category; treat them as claims. Rows marked *not in sources* are gaps this wiki will not fill by guessing.

| Dimension | Jev (System One) | LLM with JSON mode / structured outputs | Source |
|---|---|---|---|
| Output type | Typed values drawn from `criteria` you defined in advance; never a string the model composed | A string the decoder constrained to parse against a schema; your code then validates it | [[concepts/primitives]], [[reference/system-one-adapter]] |
| Probabilities | `probabilities` over every Choice option and every Score level, summing to ~1, returned on every answer | Only if you ask the model to write numbers into the JSON; TypeSafe's adapter needs a `normalize_probabilities` option because LLM distributions arrive invalid | [[concepts/confidence]], [[reference/system-one-adapter]] |
| Confidence | A 0–1 statistic derived from the distribution, on Choice and Score answers (Noul has none) | No native field; *TypeSafe's characterization*: "models tend to be overconfident and inconsistent even when prompted for a confidence estimate" | [[concepts/confidence]], [[entities/blog-introducing-system-one]] |
| Calibration | The training objective: RLCD optimises probabilities against outcomes. Calibration is a group property, never a per-answer guarantee | *TypeSafe's characterization*: RLHF optimises human preference and RLVR verifiable rewards, neither of which targets calibration | [[concepts/machine-learning-primer]] |
| Latency | Claimed 70–500 ms end to end; TypeSafe's own cookbook runs report 111 ms (14 Nouls) and 114 ms (8 Choices), measured from West Coast laptops | *TypeSafe's characterization*: 3–329 s end to end for frontier models. In the same cookbook, the LLM conditions ran 826 ms – 12,978 ms per equivalent call | [[reference/models-and-pricing]], [[cookbooks/consistency-choice]] |
| Price | $0.042 / Mtok input, output tokens free | *TypeSafe's characterization*: $0.20–$10 / MTok input with output ~5x input. The same cookbook's LLM conditions cost $0.000936–$0.041255 per call against Jev's $0.000046 | [[reference/models-and-pricing]], [[cookbooks/consistency-choice]] |
| Context | 64k tokens per request; 32k for `state` plus the longest question | **Not in sources** — no source states any LLM's context window | [[reference/models-and-pricing]] |
| Modality | Text only: string, JSON object, or array of text values. Pre-process anything else | **Not characterised in the sources** beyond "unstructured data … with an emphasis on sequential messages"; do not read a multimodality comparison into this wiki | [[concepts/state]], [[entities/blog-introducing-system-one]] |
| Hallucination / type errors | Structurally impossible to return a value outside your schema. TypeSafe's plotted 0% "is not empirical. Schema matching is guaranteed" | TypeSafe plots LLM type-error rates from OpenRouter data and flags the bias; third-party write-ups record most models between 0.58% and 13.2%, with a single 45.5% outlier | [[concepts/workflow-evals]], [[entities/press-and-coverage]] |
| Determinism / repeatability | No determinism claim, no temperature or seed parameter. Measured: mean probability std dev `0.0098` (Choice) and `0.0102` (Noul) over 15 repeats; raw top-label agreement 90.8% → 99.2% with an `uncertain` band | Measured in the same harness: "the LLM answers move from run to run, at temperature `0` too". Mean std devs `0.0245`–`0.0543` for five conditions — but Haiku 4.5 at `t=0` scored `0.0012` and 100% agreement, better than Jev | [[cookbooks/consistency-choice]], [[cookbooks/consistency-noul]] |
| Generation | None. "Not trained to generate text"; forcing it by chaining choices "will not work well and will be very slow" | The thing it is for | [[concepts/jaggedness-jev-1-13]] |
| Reasoning / indirection | Degrades with each hop of indirection; no chain of thought, no self-chosen next action | The thing RLVR reasoning models are for; the sources use them as the expensive escalation tier | [[concepts/jaggedness-jev-1-13]], [[cookbooks/sde-cascade]] |
| Failure mode when the right answer is missing | Probability still lands somewhere — Choice options always sum to 1 — so you must supply an explicit `other` / `none` route | Can invent a schema-valid but fabricated value; schema validation "catches structural errors and never semantic ones" | [[concepts/choice]], [[cookbooks/sde-cascade]] |
| Tooling | One endpoint, an OpenAPI 0.2.0 document, Python and JS SDKs at 0.6.0, an agent skill, a console playground, and an LLM adapter for comparisons | Provider-native; the adapter exists precisely so the same question code runs on either side | [[reference/http-api]], [[reference/system-one-adapter]] |

## When to pick which

| Situation | Pick | Why |
|---|---|---|
| Classify, route, rank, score, detect, or verify, at volume | Jev | Narrow answer space, code consumes the result, price and latency dominate ([[concepts/use-case-map]]) |
| The decision needs a number you can threshold and audit | Jev | Calibrated `probabilities` and `confidence` are first-class, not prose ([[concepts/confidence]]) |
| The output is prose, code, a summary, or an explanation | LLM | Jev cannot generate ([[concepts/jaggedness-jev-1-13]]) |
| The task needs several reasoning hops, or the model must choose its own next action | LLM | Explicitly outside the System One contract ([[concepts/system-one]]) |
| Input is an image, audio, or video | Neither directly — pre-process to text, then Jev | Jev is text-only ([[concepts/state]]) |
| Arithmetic, counting, date ordering | Code | Jaggedness modes 2 and 3; extract with Jev, compute in code ([[cookbooks/date-extraction]]) |
| Answer space is large but enumerable (up to 255 options) | Jev | A Choice over the enumerated options cannot invent a value ([[cookbooks/pre-parsed-value-extraction]]) |
| Answer space is open-ended text | LLM to propose, Jev to select | "Select instead of generate" ([[reference/agent-skill]]) |
| You need an explanation of *why* for a human reviewer | LLM alongside Jev | Jev returns no rationale; the distribution is the only trace you get |

## How to combine them

The cookbooks treat this as the normal case, not a compromise. Three shapes:

**Jev in front — router.** One cheap call returns `intent` (Choice) plus a `complexity` Score; code sends the easy cases to a database lookup, the hard ones to a specialist LLM, and low-confidence ones to a human. "The expensive resources only get invoked for the requests that actually need them" — [[patterns/intent-routing]], and [[cookbooks/function-calling]] for the version that also fills the handler's typed arguments.

**Jev around — guardrail.** One `system_one` call per message screens every input to and output from an LLM app: four hazard `Noul`s plus a severity `Score`, thresholded in your code into `pass`, `review`, `block`, or `support`. The same cached assessment routes differently under a strict or permissive policy because the policy lives in your code, not the prompt — [[cookbooks/llm-guardrails]]. The retrieval-side twin drops prompt injections and flags contradictions before passages reach the generator — [[cookbooks/classifying-rag-passages]].

**Jev after — judge.** A cheap LLM extracts, a Jev `Noul` battery scores P(wrong) per field with questions framed so that `true` means something is broken, and only records where a flag exceeds `0.7` get re-extracted by an expensive reasoning model. The point is stated bluntly: schema validation catches structural errors and never semantic ones; the verifier is what catches a schema-valid fabrication — [[cookbooks/sde-cascade]]. [[cookbooks/citation-check]] is the same move for citations, with a string match first and one Choice per surviving quote.

A fourth, quieter shape: **Jev as feature extractor** feeding a classical model — Score and Noul answers become numeric columns for a supervised regressor ([[cookbooks/autoresearch-feature-discovery]]).

## How to benchmark the two fairly

1. **Hold the harness fixed.** TypeSafe's own methodology is to assume the workflow code is correct and change only the model behind it ([[concepts/workflow-evals]]). Its most portable finding is not about Jev at all: every model tested was more accurate, cheaper and faster expressed as a structured workflow than as one prompt.
2. **Run both from the same question definitions.** `system-one-adapter` answers `typesafe_sdk` questions with an OpenAI or Anthropic model and returns the same `NoulAnswer` / `ChoiceAnswer` / `ScoreAnswer` objects, so nothing but the client changes ([[reference/system-one-adapter]]).
3. **Decide the LLM's answer mode deliberately, and report it.** `llm_answer_mode="probabilities"` versus `"discrete"`, and `structured_outputs=True` versus prompted JSON, change both the cost and what is comparable. TypeSafe concedes its own wrapper "tends to be slower and more expensive than giving decisions without probabilities" — that overhead is part of why its multipliers are large.
4. **Count everything the LLM side spends.** The adapter's `usage` exposes `input_tokens_total` / `output_tokens_total` across attempts, `n_retries`, `n_retries_malformed_structure`, and `latency`; a comparison that ignores corrective retries flatters the LLM.
5. **Repeat, and report variance.** The consistency cookbooks run 15 repeats per condition and report mean and max probability standard deviation plus parse-failure rate. One run of each side is not an evaluation ([[cookbooks/consistency-choice]], [[guides/testing-and-evaluation]]).
6. **Build reference labels honestly, and say what they are.** TypeSafe averages two frontier models at high thinking — which means its own score is bounded by them by construction, and that "accuracy" means agreement, not correctness. If you have real outcome labels, use those instead.
7. **Check the contract before you publish.** MCA §2.3(f) prohibits publishing "benchmarks or performance information about the Services." Running private evals is exactly what TypeSafe's own anti-benchmaxxing post recommends; publishing them is a separate question for your legal team ([[reference/legal-and-data]], [[entities/blog-antibenchmaxxing]]).

## Caveats on TypeSafe's own claims

- **"193.6x faster, 444.6x cheaper"** is Jev against LLMs inside TypeSafe's harness, wrapped by TypeSafe's own constrained-output wrapper, scored against labels generated by two competitors' models — and TypeSafe says the figures "are on the higher end of real world gains" ([[concepts/workflow-evals]]).
- **"Zero hallucinations"** means schema conformance only, and the 0% in the plot "is not empirical." The judgment can still be wrong ([[entities/jev]]).
- **The LLM comparison rows above are TypeSafe's characterization of the competition**, not measurements this wiki can verify. The one place the sources measure both sides side by side — the consistency cookbooks — includes a result that favours an LLM (Haiku 4.5 at temperature 0), which TypeSafe publishes and captions "100% repeatability does not imply correctness."
- **Speed numbers include TypeSafe's network path**: "our published evals are generally run from our laptops on the West Coast."
- **The press adds no independent data.** Outlet figures differ only by aggregation — best workflow row (Jev 76.0%) versus four-workflow average (Jev 67.8%) — and all trace to the same vendor evals ([[entities/press-and-coverage]]).
- **RLCD is undisclosed.** No paper, no published calibration curves, no reproducible training description exists in any captured source; one write-up notes prior art on rewarding calibrated confidence ([[entities/press-and-coverage]]).

## Related

- [[concepts/system-one]] — the model class and TypeSafe's own comparison table
- [[concepts/confidence]] — the field that has no JSON-mode equivalent
- [[concepts/jaggedness-jev-1-13]] — what Jev cannot do, in detail
- [[reference/system-one-adapter]] — running the same questions on an LLM
- [[concepts/workflow-evals]] — the methodology behind every multiplier quoted here
- [[guides/testing-and-evaluation]] — evaluating your own workflow
- [[syntheses/faq]] — short-form answers
- [[syntheses/glossary]] — the terms used above

## Sources

- wiki pages listed in the frontmatter (all under `wiki/`)
- raw/site/blog-introducing-system-one.txt (https://typesafe.ai/blog/introducing-system-one-models-and-jev) — "Frontiers, Old and New" table, evidence and nuance sections
- raw/site/typesafe-ai-home.txt (https://typesafe.ai/) — the unanswered homepage FAQ question this page addresses
- raw/github/system-one-adapter-python/README.md (https://github.com/typesafe-ai/system-one-adapter-python) — provider structured-output mechanics and usage accounting
- raw/docs/cookbooks__consistency_choice_cookbook.md, raw/docs/cookbooks__consistency_noul_cookbook.md (https://docs.typesafe.ai/cookbooks/) — the side-by-side latency, cost and variance tables
- raw/site/typesafe-ai-legal_mca.txt (https://typesafe.ai/legal/mca) §2.3(f) — benchmark-publication restriction