--- title: "Workflow evals: how TypeSafe measures Jev" type: concept tags: [evals, benchmarks, methodology, claims, workflows] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/site/evals-site.txt - raw/site/evals-security_incidents.txt - raw/site/evals-agent_trace_observability.txt - raw/site/evals-invoice_processing.txt - raw/site/evals-customer_service.txt - raw/site/blog-introducing-system-one.txt jev_version: "jev-1.13.0" summary: "TypeSafe's four workflow evals: same harness for every model, reference labels from GPT-6 Astra + Claude Fable 5.1, source of 193.6x/444.6x." --- # Workflow evals: how TypeSafe measures Jev > **TL;DR** TypeSafe fixes a code workflow, runs every model through the same harness, and scores each model against reference labels produced by averaging **GPT-6 Astra** and **Claude Fable 5.1** at high thinking. Across four workflows Jev is claimed **193.6x faster and 444.6x cheaper**; TypeSafe itself says these "are on the higher end of real world gains." Treat the numbers as vendor-run, self-labeled, and useful mainly as evidence for the *architecture* claim: a structured workflow beats a single prompt for every model tested. ## What it is A vendor-published eval suite at `https://evals.typesafe.ai/` with four end-to-end automation workflows. Its framing sentence: "Real world tasks can be executed via **structured workflows** or **standalone prompts**. Structure is always better." The headline plot averages "one model configuration's accuracy, cost and time over the four workflows with equal weight, against the consensus labels. Every model runs at its provider's default reasoning setting. Up and to the left is better." Model families plotted: **TypeSafe, OpenAI, Anthropic, Fireworks**, each in two variants — `workflow` and `prompt`. Two Pareto frontiers are drawn: "nothing is both cheaper and more accurate" and "nothing is both faster and more accurate." ## Methodology ### 1. Decompose the work, build a harness "To automate a task, we decompose the decisions into programmatic rules and intelligent judgments. Rather than ask a model to solve the entire problem in one shot (like the prompt examples in the plot), we ask independent narrow questions and defer to code where possible." The three question types used, as the eval site describes them: | Type | Question | What the model returns | |---|---|---| | **Noul** | yes or no | a probability | | **Choice** | one option from a defined set | a distribution over the choices, as well as confidence | | **Score** | a continuous rating on a scale | a score, distribution over the levels, and confidence | The results are then used programmatically to produce the output actions. **The key finding:** "Averaged across the four example tasks, every model is more accurate, cheaper and faster in the workflow than it is with the same policy as a prompt." This is the most portable result on the site — it is about harness design, not about Jev. The site's toy illustration is an expense-claims policy, where each sentence becomes either a typed question or a code rule: 1. Every claim comes with a receipt. If the receipt cannot be read, ask the employee for a new one. 2. Work out what kind of expense it is: a meal, travel, or equipment. 3. A meal over $75 needs a manager's sign-off when the description on the claim does not clearly match the receipt. 4. Everything else is approved. ### 2. Assume the harness is correct "Instead of debating the correctness of the harness and labels, we assume that the code is correct, and measure against the current smartest large models. For this eval, the reference labels are generated via an average of the responses of **GPT-6 Astra and Claude Fable 5.1, both at high thinking**, answering every question in the harness. All other models are evaluated using the provider's default reasoning settings." Restated in the blog: "every model gets the same workflow. We test how they compare to the average of the smartest models (in this case, Astra and Fable)." TypeSafe explicitly does **not** optimize for a ground-truth classification and does not allow the harness or model to change — "potentially allowing for overfitting via harness engineering." ### 3. Wrap the LLMs so they can answer the same questions "The LLMs use our **System One LLM** wrapper, which constrains LLMs to output structured decisions compatible with our API. We have found this to be the most accurate way to get decisions from LLMs, but this tends to be slower and more expensive than giving decisions without probabilities." Each workflow page also publishes a handful of qualitative cases: "for each model, one where it alone differs from the other two; one where all three miss the reference; one where all three agree." The three models walked through on those pages are labelled **TypeSafe**, **Opus**, and **Sol**. ## The four workflows | Workflow | Task | Inputs | Output | |---|---|---|---| | **Security Incidents** | "A security alert fires on a laptop or a server. Given the alert and everything on file about that machine, we decide whether to close it, pass it to an analyst, or contain it now." | The alert (what fired, on which asset, when); the asset (environment, tier, owner); open tickets; registered devices; scheduled maintenance; standing authorizations | **one of** `AUTO CLOSE`, `NOTIFY USER`, `ESCALATE TIER2`, `KILL PROCESS`, `DISABLE ACCOUNT`, `ESCALATE URGENT` | | **Agent Trace Observability** | "A support agent has just finished with a customer. Given the whole run, every tool call included, we decide whether a person needs to look at it, and how soon." | The agent's instructions; the conversation; tool calls with arguments and results; the final message; customer feedback when left | **one of** `AUTO-CLOSE`, `NOT A BUG`, `HUMAN REVIEW`, `PRIORITY REVIEW`, `FILE ISSUE · ROUTE`, `PAGE ON-CALL` | | **Invoice Processing** | "A vendor's bill arrives. Given the bill, the order behind it, and what was actually delivered, we decide whether it gets paid, held, or sent back." | The invoice; the purchase order; the contract; the vendor record; prior invoices; correspondence; delivery evidence; approvals | **any that apply**: `PAY`, `SCHEDULE`, `SHORT PAY`, `ROUTE FOR APPROVAL`, `HOLD FOR DOCUMENTS`, `REQUEST CORRECTED INVOICE`, `DISPUTE LINES`, `FRAUD REVIEW`, `DUPLICATE` | | **Customer Service** | "A customer writes in. Given the thread so far and the state of their account, we decide what the assistant should say and do next." | The conversation; the customer record; the account (subscription, recent charges, refunds, card status); a pending proposal | **any that apply, sometimes none**: `SAY`, `REFUND`, `FREEZE CARD`, `SET INTENT`, `HAND OFF`, `FLAG FOR REVIEW`, `CLOSE` | ### Workflow structures (worth copying) **Security Incidents** — four stages. (1) Read the alert: three questions about the alert and the records joined to it — was the activity unauthorized, does a record explain it, how strong is the evidence. (2) Close, queue, or act: code combines the three answers with how important the machine is and where it runs; borderline identity alerts also notify the user. (3) The state of the incident: acting opens **eleven more questions** — credentials, live sessions, mail, anything left behind to run later, processes, network traffic, and how far the activity spread. (4) Choose the response: the playbook takes the first group whose conditions are met, then the strongest step in it that still applies; when no group applies, the alert is escalated. **Agent Trace Observability** — three stages. (1) Permission on irreversible actions: every action the agent took that cannot be undone is measured against what it was permitted to do at that point in the run; a breach pages on-call and the review ends there. (2) Task completion and user satisfaction, asked as two separate questions reading different parts of the run "so neither colours the other." (3) The two answers select one of four outcomes — healthy, an expectation gap, an overt failure, or a silent failure — and each asks one further question and then ends the review: close, queue, file a bug, or send to a person. **Invoice Processing** — four stages. (1) Read the documents: seven rounds of questions, one subject at a time; "sums, dates, account numbers and statuses are computed in code rather than asked." (2) Reasons to stop: four conditions end the review immediately — a fraud pattern, an invoice already paid, the wrong vendor or company, or a document that is not an invoice. (3) Holds, disputes and corrections: three rule lists run side by side, every rule that applies adds its actions. (4) Releasing the payment: a sequence of yes-or-no checks decides how the payment leaves. **Customer Service** — four stages. (1) Intent, frustration and risk flags: **eleven readings of the conversation at once** — what the customer wants and how they want it resolved, how frustrated they sound, how urgent it is, and flags such as unauthorized activity, a legal threat, or a request for a person. (2) Consent, fraud, money, retention: four follow-ups plus a tie-break when two intents are close, each running only when the first reading calls for it, with the records it needs attached. (3) Check the assistant's claims against the record — "did it tell the customer a refund had gone through, or that the card was frozen, when neither had happened?" (4) Nine sections run top to bottom: safety, the assistant's claims, a pending proposal, handing off, closing, then the issue itself; the first rule that applies in a section emits its actions, otherwise the next section runs. Common design signature across all four: **many atomic questions asked at once, deterministic arithmetic and status checks kept in code, early-exit stop conditions, and ordered rule groups where the first match wins.** ## What "193.6x faster, 444.6x cheaper" means From the blog: "This is where the claims of 193.6x faster, 444.6x cheaper on our home page comes from." - The comparison is **Jev vs. the LLMs running the identical workflow harness**, averaged across the four workflows above — not Jev vs. a raw LLM API call, and not a benchmark of intelligence. - The LLM baselines run through the System One LLM wrapper, which TypeSafe says "tends to be slower and more expensive than giving decisions without probabilities" — the wrapper is part of what makes the multiplier large. - TypeSafe frames accuracy separately: "Jev is off the charts – owning the Pareto frontier for almost 2 orders of magnitude." - TypeSafe's own bound on the number: "we expect that these are on the higher end of real world gains." ### Caveats TypeSafe publishes (verbatim in substance) 1. **Authorship bias.** "These content of these workflows were not deliberately chosen nor constructed to make our model look good, and are not in our training distribution. However, they were made by individuals on our model capabilities team, so some bias could exist." 2. **Reference-label bias.** "We use the average of GPT-6 Astra and Fable 5.1 as the reference answer, which biases answers towards OpenAI and Anthropic's models. We likely underestimate the relative performance of our model and DeepSeek's models." Note the direction: the scoring reference *is* two of the competitors, so Jev can never score above the reference by construction. 3. **Wrapper overhead.** The LLMs are constrained through TypeSafe's own System One LLM wrapper (above). 4. **Reasoning settings are asymmetric.** Reference labels use Astra and Fable "both at high thinking"; every evaluated model runs at "the provider's default reasoning setting." 5. **Measurement location.** "Our published evals are generally run from our laptops on the West Coast (this is where our service is currently based)" — latency numbers include that network path. 6. **Complexity is deliberate.** "Note that the calls here are significantly more complex than the side-by-side demonstration above. That's because they're more representative of the types of production workloads needed for true business automation." ### Separate claims that are *not* from the workflow evals - The **side-by-side demo** used **GPT-5.6 Terra** with default reasoning, "because we've found it to be the most comparable at intelligence to Jev on average," on a deliberately short, dense state — TypeSafe notes "the relatively shorter input paints our model in an advantageous light," and that the only disagreement in the recorded run was on "Churn likelihood level," which TypeSafe calls genuinely ambiguous. - The **hallucination / type-error plot** uses OpenRouter numbers for LLMs ("there almost certainly is bias here: more complex queries might be routed to better models"), and Jev's 0% is **not empirical**: "Schema matching is guaranteed, thus we can confidently add 0% into the plots." - The **Wikiracing demo** ran against non-reasoning modes of the models (except Astra at its lowest reasoning setting), so "our speedups here tend to be a lot less than in previous demos." ## Why it matters for code - **Copy the harness shape, not the score.** The transferable result is that the same policy expressed as a workflow beat the same policy expressed as a prompt for *every* model tested. - **Ask questions in bulk.** Three of the four workflows front-load a large parallel batch (eleven questions in two of them) and only then branch. See [[patterns/fan-out]]. - **Keep arithmetic in code.** Invoice Processing explicitly computes sums, dates, account numbers and statuses in code rather than asking the model. - **Early exits are cheap correctness.** Both Invoice Processing and Agent Trace Observability end the review immediately on a stop condition rather than scoring everything first. - **Build your own reference set the same way** if you lack labels: an ensemble of expensive reasoning models answering every question in your harness. See [[guides/testing-and-evaluation]] and [[cookbooks/autoresearch-feature-discovery]]. ## Gotchas - These are **vendor-run evals with vendor-generated labels**. There is no independent ground truth and no third-party replication in the sources. - "Accuracy" here means **agreement with the Astra+Fable average**, not correctness. - The four workflows are the published set; the site notes deeper material (examples, disagreements, full queries) lives on `evals.typesafe.ai` itself, which the captured text only summarizes. - Per-workflow numeric results are not present in the captured text — the plots are images. Do not quote per-workflow accuracy figures from this page; only the aggregate 193.6x / 444.6x is textually attested. ## Related - [[concepts/system-one]] — the claims this eval backs - [[concepts/how-to-build]] — the harness design the evals embody - [[patterns/fan-out]] — bulk parallel questions - [[patterns/confidence-routing]] — probability-driven branching - [[guides/testing-and-evaluation]] — evaluating your own workflow - [[entities/blog-introducing-system-one]] — the launch post - [[entities/blog-antibenchmaxxing]] — TypeSafe on benchmarks - [[reference/models-and-pricing]] — the price and latency the evals rest on ## Sources - raw/site/evals-site.txt (https://evals.typesafe.ai/) - raw/site/evals-security_incidents.txt (https://evals.typesafe.ai/security_incidents.html) - raw/site/evals-agent_trace_observability.txt (https://evals.typesafe.ai/agent_trace_observability.html) - raw/site/evals-invoice_processing.txt (https://evals.typesafe.ai/invoice_processing.html) - raw/site/evals-customer_service.txt (https://evals.typesafe.ai/customer_service.html) - raw/site/blog-introducing-system-one.txt (https://typesafe.ai/blog/introducing-system-one-models-and-jev) — "Evidence / Technical Results"