$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/ideas

Measurements, access routes and open replicas

[ community tier ][ updated 2026-09-21 ][ confidence medium ][ jev-1.13.0 ]#measurements · benchmarks · access · replicas · community

TL;DR The numbers behind the verdicts in Field reports: independent evaluations, critiques, open replicas: per-call cost and latency people measured, throughput, accuracy on small sets, gateway routes, and trust notes on open replicas. Every figure is one person's run on their own workload; re-measure before relying on any of it (Testing and evaluating a Jev workflow).

Measured numbers

Never quote these as facts. Each is one poster's — or one vendor's own — run.

Number Reported by Setup Caveat
$0.00035/call, 0.44 s avg; $0.34 total vs $28.17 for Claude Sonnet 4.6; mean per-case variance 0.0000149 @LangChain Deep Agents weather agent, 5 fixed traces, 100 reps/judge, human oracle, langchain-typesafe==0.0.1a2 5 cases; Jev service version not recorded
112x lower model cost than GPT-6 Astra (code-execution computer use); 245x vs screenshot computer use; WebMCP cut cost a further 18% @0xidanlevin WindTunnel, 49 tasks × 8 sites, Jev + Mercury 2.5 Open, reproducible; compares harnesses, not models
1,284 tool calls in one session, 938 yes/no; bill $64.77 → $26.16; $0.000041/decision; 38 ms vs 1.84 s @0xCarnagee One session, replayed side by side Promotional, single run
100,000 X posts in 20.4 s for $0.67, 14 Nouls each; Claude Opus 5 did 214 posts for $0.98 on the same clock @0xMovez Self-built analyser, one parallel pass Opus leg is truncated and extrapolated to $458, not equal work
Classify 138.6 ms vs 794.3 ms median (5.73x); split 209.6 ms vs 1,352.3 ms (6.45x), Jev 1.13.0 vs GPT-5.6 Luna, decision time only @jerryjliu0, DocJev README 40 public-finance PDFs, 8 packets, text pre-extracted by LiteParse Small pilot; OCR time excluded from these medians; see Patterns: judging, search, documents, real-time and markets P26
1,700 real emails, ~4M input tokens, 18¢; ~200 ms/query; 4 questions each (category, priority, spam, reply) Ryan Vogel via @startupideaspod personal inbox, one pass Arithmetic consistent with $0.042/Mtok (4M ≈ $0.17); latency self-reported
50 calls at 25 concurrent: 741 ms, no 429s; all 50 at once was slower (~1 s). Claim check on 16 labelled headlines: true 0.83-0.90, false 0.01-0.12 Nate B. Jones run through OpenRouter One run; 25-way concurrency is a starting point, not a limit
$0.00115 vs $0.039 per page and ~0.5 s vs ~3.3 s against a Claude Sonnet classifier; 0 wrong on 1,067 tax-form pages, 38 under a 0.95 gate @nedwize, kyotofin/tax-doc-classifier open repo, reproducible eval His corpus; blank and benchmark forms, not messy scans
~30k tokens in ~160 ms; server time barely moved up to ~100 questions, then rose steadily; 1,500 questions still returned in a few hundred ms; a 200-option question was as quick as a 2-option one Archer Hume, jev-1.13.0, 2026-09-17 192 + 148 sequential requests, shuffled, 8 repeats per size Times are the x-envoy-upstream-service-time header: shared service, queueing included, not a hardware benchmark
Token accounting is additive: 268 input tokens for one minimal Noul, 276 for two, 318 for a Noul + 2-option Choice + 2-level Score; a 23k-token state with 5,000 questions was accepted same 311 accounting + 35 limit-boundary requests verified against the 64k-per-request / 32k-per-branch budgets with state counted once (Models, aliases, pricing, rate limits, context)
Calibration: 10-bin ECE 0.031 over 1,200 MMLU items (990 of them in the 0.9–1.0 bin); MMLU-Pro 84.6%; fresh 3-digit multiplication 86.7% correct at mean top probability 0.83; two-step word problems 32% at 0.30; modular exponentiation 56% at 0.35 same 6,800 benchmark records + 190 generated maths items Recomputed from recorded probabilities, not the API confidence field. Family averages can hide bin-level error; maths stays a documented weakness (Jev 1.13 jaggedness: known failure modes §2)
Option-set effects: reversing a ticket's options moved the top probability 0.84–0.89 → 0.93–0.96; adding one irrelevant option moved the log-odds between two existing options +0.38 → +0.11 (mean −0.28, paired 95% ≈ −0.36…−0.19, lower in all 10 blocks) same 181 order + 96 reference-card + 50 block-randomised requests unverified against docs — one scenario per template; advisory reading in Field reports: independent evaluations, critiques, open replicas
output_tokens is billing arithmetic, not decoding: a Noul request costs 4 shared tokens + 15 per answer + the token length of each question id (which the docs say the model never sees); a 255-option response reported 2,714 same 311 requests Output tokens are free (Models, aliases, pricing, rate limits, context); never divide them by request duration and call it speed
Jev's tokenizer matches none of 192 public tokenizers across 415 probes (closest Qwen at 348/415); every digit is a separate token; 8 as count as 1 token, 16 as 4 same 445 requests unverified. Local tiktoken or character-count estimates drift in both directions — leave headroom under the caps
100,000 compliance alerts: >$290K on Opus 5 vs <$26K through the harness; 50–95% saving claimed for workflows vs a frontier-model agent generally; ~$3 per alert for a frontier agent ($2.89 on their production agent → $0.25 per alert through the tuned workflow) Grep.ai's AgentRun, @MiguelriosEN and @_aj one AML alert-review job: 100-alert arm comparison, then a 1,000-alert tuning run The vendor's own figures about its own product, one regulated domain. Pattern: Patterns: agent internals, context and coding agents P37
By the end of the run 98 of every 100 cases went through the workflow (hand-back ~1 in 70 over the last 300); their compliance judge scored 90% on the first 100 full-agent alerts and 95% over the last 300; one alert fell from 826 tool calls / 51 min to ~30 calls / ~3 min; ~30 Jev questions per alert cost about a third of a cent; 868 hits re-judged by replay in 2 minutes for a few cents; the tuning run cost $1,049 same same Same caveat. The accuracy grader is their own LLM judge against historical labels, not a fresh human relabel
"About 150 ms" per typed answer, at a few hundred-thousandths of a dollar same vendor description Consistent with $0.042/Mtok and the 70–500 ms band others report; not independently measured

TypeSafe's own 193.6x / 444.6x is a workflow-average ceiling (Workflow evals: how TypeSafe measures Jev). @cyrilXBT calls them "a ceiling, not a typical result": the multiplier comes from a workflow's decision-to-generation ratio, not a head-to-head call.

Open replicas and Jev-like alternatives

What each one is and how it works: Repos: data, documents, judging, real-time, markets, business apps, replicas. Trust notes only here.

For a build decision. The replicas show the interface is commodity — one forward pass, read scores over a closed option set — so the API shape is not lock-in and porting later is realistic. They do not show that the calibration is commodity: mini-Jev says outright its scores are not calibrated probabilities, and Confidence-gated routing rests on calibration (Confidence vs probability). So prototype on the hosted API, keep questions and thresholds in your own code, and self-host only once a labelled set proves a local model matches on your decisions. Replica claims are unverified too.

Access, waitlist and practicalities

All unverified; confirm against Models, aliases, pricing, rate limits, context and console.typesafe.ai (console + playground) before relying on any of it.

Related

Sources

Links are inline in each row. Raw captures (raw/x/, raw/x-repos/, raw/nate/, raw/community/) live in the private repo.