~/wiki/ideas
Field reports: independent evaluations, critiques, open replicas
TL;DR Independent tests (2026-09-16→21, all narrow, none peer-reviewed) agree on one shape: Jev is excellent at bounded, pre-enumerated decisions over a state you already filtered, and mediocre when it must build its own option menu or act over many steps. Cost and repeatability wins are real; "200x/400x" is a workflow ceiling. Re-measure on your own data (Testing and evaluating a Jev workflow).
What held up
| Finding | Who | Evidence quality | Note |
|---|---|---|---|
| Repeatability far above LLM judges: quality-score variance 92–913x lower than GPT-5.6 Luna/Terra and Claude Sonnet 4.6; binary pass/fail matched a human oracle 500/500 (Terra 99.8%, Luna 96.4%, Claude 80.0%) | @LangChain (Shea & Roche) | measured, repo published | verified in direction — docs claim "extremely consistent" (Jev 1.13 jaggedness: known failure modes §8). 5 traces × 100 reps; does not generalise. |
| Tool selection from a menu the app already built is the sweet spot | @0xidanlevin, @trycua | measured | Matches Intent routing, jaggedness §4. |
| Parallel questions are nearly free; add questions, not round trips | @sydneyrunkle | anecdotal | verified — one state ingest, 64k/32k budgets (Speculative fan-out). |
| Output can be wrong but never malformed | @akshay_pachaar | anecdotal | verified — "Schema matching is guaranteed" (Workflow evals: how TypeSafe measures Jev). Not "cannot be wrong". |
| Compaction by relevance-scoring tool calls (survivors kept verbatim) ships | @0xCarnagee on fast-jev-compaction | anecdotal | Caveats below. |
Question isolation is real: a secret in a sibling question was invisible (p 0.00); the same line in state read 0.90–0.92 |
Archer Hume, ~10,000 API calls | measured, payloads published | verified (Primitives: Choice, Score, Noul). Shared facts belong in state, never in another question. |
| Injected fake options never displaced the real ones (105 requests, 7 delimiter formats) | same | measured | verified — an option outside criteria can never be returned (Choice questions). State text can still shift probabilities (jaggedness §6). |
| Calibration held on 1,200 MMLU items (ECE 0.031), and probability fell on the harder fresh-maths task | same | measured, numbers in Measurements, access routes and open replicas | verified in direction (AI primer: why calibrated decision models); 990 of 1,200 sat in the top bin, so the check is thin below it. |
| Batching holds far past the doc examples: ~100 questions before server time moves, 1,500 still a few hundred ms, 5,000 accepted beside a 23k-token state | same | measured, server-time header | verified — state is ingested once under the 64k/32k budgets (Models, aliases, pricing, rate limits, context). |
confidence is arithmetic over probabilities, not a second learned estimate: TypeSafe's adapter uses (p_max − 1/K)/(1 − 1/K) |
same, reading confidence_metrics.py |
code read | verified for system-one-adapter: LLM-backed drop-in for TypeSafeClient; unverified for the hosted API — Confidence vs probability says the formula is unpublished. A sharp distribution can still be confidently wrong. |
| Explainer threads mostly restate launch numbers | — | restatement | Not evidence; idea lists at best. |
One person's inference, not TypeSafe's description. Archer Hume reconstructs Jev from those probes as a causal transformer ending in a readout instead of a decode loop: state held once as a shared prefix, each question an isolated suffix branch, the options read as one list before a single decision position, probably a sparse (MoE) backbone. TypeSafe has published only the parallel-output and RLCD claims (System One Models); the rest is black-box deduction, as he says himself. A mental model, not a contract.
Where it disappointed or broke
| Finding | Who | Evidence quality | Failure condition |
|---|---|---|---|
| Standalone browser control solved only 25/49 tasks; WebMCP tool menus took it to 49/49 | @0xidanlevin | measured, open repo (WindTunnel) | Jev must pick the right next step, not just a valid control. Author's framing: their harness, not a limit on Jev. |
| Cannot generate tool arguments; needed Mercury 2.5 beside it for search queries | same | measured | verified — jaggedness §9. Budget for a second small model. |
| Form-fill decision accuracy 83.6% hosted vs 99.7% for a task-trained local specialist | @trycua | measured, narrow | The specialist was fine-tuned for that exact convention. Decision-level, not end-to-end. |
| Hosted round trip 260–280 ms vs 7–9 ms local | same | measured, different boundaries | Network is the floor; no sub-100 ms loops. |
| Four overlapping Choice options ("high priority"/"urgent"/"needs review"/"flag") gave confident answers that were internally inconsistent "about a third of the time" | @cyrilXBT | anecdotal, self-reported (~6 h to fix) | verified as a schema bug, not a model bug — jaggedness §1, §7. Options must be mutually exclusive. |
| A 0.6B open replica beat Jev on ViZDoom Basic (128/128 vs 56/128), tied on Predict Position (11/128) | TianyuCodings/NanoJev | unverified — replica-run, Jev-side harness unchecked |
Spatial/game control is not a System One strength; a task-trained tiny model can win. |
| Too expensive for high-frequency control. $0.042/Mtok is ~7x a DeepSeek V4.1 Flash cache read; 10 Hz at 10k context ≈ $15/h | @jpschroeder | measured arithmetic | Price verified (Models, aliases, pricing, rate limits, context); arithmetic checks out. He wants ~10x cheaper, on-device, for robotics. |
| Jev-scored compaction caveats, per the plugin's own README | tamaratran/fast-jev-compaction | author-stated | Full state re-sent per request, so long history costs a request per handful of questions; sizes are character estimates, not a tokenizer; "a probability is not a proof that a result is safe to delete"; falls back to the built-in summary on failure. |
/compact timing needed a private, hand-labelled eval set (40 sessions) plus prompt hill-climbing |
@kunchenguid | anecdotal | Thresholds are yours to find; he shifts precision→recall as the window fills. |
| Per-minute BTC buy/hold/sell signal "performed poorly"; GPT-6 Astra did slightly better via news. Spelling an answer letter by letter was "rough" | Ryan Vogel via @startupideaspod | anecdotal | verified direction — time series and generation are documented weaknesses (Jev 1.13 jaggedness: known failure modes). He keeps Jev advisory, off portfolios. |
| Packet splitting: 7/8 packets exact vs 8/8 for GPT-5.6 Luna; classification tied at 40/40 | jerryjliu/docjev | measured, open repo, 40 real PDFs | Boundary detection is the weaker half; gate low-confidence boundaries to review. |
| Keyword/regex rules vs Jev on tricky small sets: support routing 4/12 vs 11/12 (miss at 0.53 confidence, hits ≥0.75); changelog "major release" 1/8 vs 8/8 | Nate B. Jones | measured by the guide's authors, tiny sets | Not production accuracy. The useful part is the confidence gap between hits and misses. |
Score 1-10 on headline quality ordered items sensibly but rated the author's own published headlines 5-7 |
same | measured | Rank with a score; not a pass mark. |
Options 16GB and 16 GB offered separately split the probability; "is this laptop working?" missed a listing that mentioned a liquid-spill repair |
same | measured | verified — overlapping options and literal reading (Jev 1.13 jaggedness: known failure modes). Dedupe options; add the question that would have caught it. |
The option list itself moves probabilities. Reversing a ticket's options shifted the top probability ~0.84–0.89 → 0.93–0.96 (a reference option read 16/16 last, 12/16 first, 11/16 middle, 48/48 from state); adding one irrelevant option shifted the log-odds between two existing ones +0.38 → +0.11, lower in all ten blocks |
Archer Hume | measured, ~330 requests | unverified — not in the docs, and it qualifies "give the full list, not a shortlist" (Choice questions). A 0.9 gate can flip on a permutation; shuffle options in your eval and re-tune when the set changes. |
| Do not assume determinism: identical requests returned slightly different probabilities, duplicate questions in one request included; response key order varied | same | measured | unverified — docs promise "extremely consistent" (Jev 1.13 jaggedness: known failure modes §8), not bit-exact. Don't key caches or tests on exact values. |
output_tokens is a billing figure, not generation, and the tokenizer matches no public one, so local estimates drift |
same | measured; figures in Measurements, access routes and open replicas | Never read decode speed or generation out of it; leave headroom under the 32k/64k caps. |
| AgentRun's own caveats: crash recovery does not cover model nodes or nested workflows yet; on one task the author matched every development label and scored no better on the reserved set | Grep.ai, @MiguelriosEN | vendor self-report | Keep a held-out set the author never sees. |
No post in this corpus reports a reproducible correctness failure of Jev itself; the failures above are schema design, missing generation, harness scope, latency, price, and — new from the architecture probes — sensitivity to option order and option-set composition.
What this means when advising on a project
- Re-measure on the project's own data (method: Testing and evaluating a Jev workflow); every number here is n≈1.
- Expect the workflow multiplier, not the headline one: gains scale with the decision-to-generation ratio.
- Prefer narrow, mutually exclusive questions, and treat the option list as tuning surface: overlapping options produced confident nonsense, and order or an added option moved probabilities. Shuffle options in your eval; re-check thresholds when the set changes.
- Keep Jev out of anything that must generate (tool arguments, queries, summaries); pair it with a small fast LLM.
- No control loops faster than ~4 Hz or under a 250 ms budget (260–280 ms round trips); at high frequency, price it per hour, not per decision.
- Don't ship Jev-scored compaction without an eval set and a fallback; both published tools have one.
- Watch benchmark circularity: TypeSafe scores against an Astra+Fable average; replicas score against TypeSafe's records.
- Model routing may not pay for itself mid-session: Diogo Almeida (TypeSafe's CEO) works the arithmetic and finds routing to a cheaper model and back can cost more, because the large model re-processes the context — on his assumed mix, pure large-model was about two-thirds the cost (post). See Patterns: agent internals, context and coding agents P02.
Related
- Measurements, access routes and open replicas — the numbers behind these verdicts; Consult guide: could Jev help this project? — the project walkthrough
- Decision patterns from the community (with fit verdicts) — community patterns with fit verdicts; Community repos: what people built and how they use Jev — the projects tested
- Jev 1.13 jaggedness: known failure modes — the official failure-mode list these confirm; Workflow evals: how TypeSafe measures Jev — where 193.6x / 444.6x comes from
- Testing and evaluating a Jev workflow — running your own measurement
Sources
Posts, repos and essays listed in this page's frontmatter (raw/x/, raw/x-repos/, raw/community/, raw/nate/); URLs inline above. The probe study is raw/community/archerhume-jevs-architecture-unmasked.md (https://archerhume.com/posts/jevs-architecture-unmasked), 2026-09-17.