---
title: "Field reports: independent evaluations, critiques, open replicas"
type: community
source_tier: community
tags: [field-reports, evaluations, critiques, replicas, benchmarks]
created: 2026-09-20
updated: 2026-09-21
confidence: medium
sources:
  - raw/community/archerhume-jevs-architecture-unmasked.md
  - raw/community/diogo-almeida-coding-agent-notes.md
  - raw/x/MiguelriosEN-2101029313906987422.md
  - raw/x/_aj-2102061534956662818.md
  - raw/nate/jev-shaped-problems.md
  - raw/x/LangChain-2101454284927959080.md
  - raw/x/0xidanlevin-2100937437325205568.md
  - raw/x/trycua-2101437979180904640.md
  - raw/x/jpschroeder-2100952337904767258.md
  - raw/x/cyrilXBT-2101515130798297448.md
  - raw/x/0xCarnagee-2101456270909690202.md
  - raw/x/kunchenguid-2101032677940117875.md
  - raw/x/startupideaspod-2101029801821941933.md
  - raw/x-repos/jerryjliu__docjev.md
  - raw/x-repos/TianyuCodings__NanoJev.md
  - raw/x-repos/tamaratran__fast-jev-compaction.md
  - raw/x-repos/nekuda-ai__WindTunnel.md
jev_version: "jev-1.13.0"
summary: "What independent testers measured in Jev's first week: repeatable bounded decisions, weak standalone browser control, and probe-measured order and option-set sensitivity."
---

# Field reports: independent evaluations, critiques, open replicas

> **TL;DR** Independent tests (2026-09-16→21, all narrow, none peer-reviewed) agree on one shape: Jev is excellent at **bounded, pre-enumerated decisions over a state you already filtered**, and mediocre when it must build its own option menu or act over many steps. Cost and repeatability wins are real; "200x/400x" is a workflow ceiling. Re-measure on your own data ([[guides/testing-and-evaluation]]).

## What held up

| Finding | Who | Evidence quality | Note |
|---|---|---|---|
| Repeatability far above LLM judges: quality-score variance **92–913x lower** than GPT-5.6 Luna/Terra and Claude Sonnet 4.6; binary pass/fail matched a human oracle **500/500** (Terra 99.8%, Luna 96.4%, Claude 80.0%) | [@LangChain](https://x.com/LangChain/status/2101454284927959080) (Shea & Roche) | measured, repo published | `verified` in direction — docs claim "extremely consistent" ([[concepts/jaggedness-jev-1-13]] §8). 5 traces × 100 reps; does not generalise. |
| Tool **selection** from a menu the app already built is the sweet spot | [@0xidanlevin](https://x.com/0xidanlevin/status/2100937437325205568), [@trycua](https://x.com/trycua/status/2101437979180904640) | measured | Matches [[patterns/intent-routing]], jaggedness §4. |
| Parallel questions are nearly free; add questions, not round trips | [@sydneyrunkle](https://x.com/sydneyrunkle/status/2100754364545761643) | anecdotal | `verified` — one `state` ingest, 64k/32k budgets ([[patterns/fan-out]]). |
| Output can be wrong but never malformed | [@akshay_pachaar](https://x.com/akshay_pachaar/status/2101309986156712025) | anecdotal | `verified` — "Schema matching is guaranteed" ([[concepts/workflow-evals]]). Not "cannot be wrong". |
| Compaction by relevance-scoring tool calls (survivors kept verbatim) ships | [@0xCarnagee](https://x.com/0xCarnagee/status/2101077261407412732) on fast-jev-compaction | anecdotal | Caveats below. |
| Question isolation is real: a secret in a **sibling question** was invisible (p 0.00); the same line in `state` read 0.90–0.92 | [Archer Hume](https://archerhume.com/posts/jevs-architecture-unmasked), ~10,000 API calls | measured, payloads published | `verified` ([[concepts/primitives]]). Shared facts belong in `state`, never in another question. |
| Injected fake options never displaced the real ones (105 requests, 7 delimiter formats) | same | measured | `verified` — an option outside `criteria` can never be returned ([[concepts/choice]]). State text can still shift probabilities (jaggedness §6). |
| Calibration held on 1,200 MMLU items (ECE 0.031), and probability fell on the harder fresh-maths task | same | measured, numbers in [[ideas/measurements]] | `verified` in direction ([[concepts/machine-learning-primer]]); 990 of 1,200 sat in the top bin, so the check is thin below it. |
| Batching holds far past the doc examples: ~100 questions before server time moves, 1,500 still a few hundred ms, 5,000 accepted beside a 23k-token state | same | measured, server-time header | `verified` — `state` is ingested once under the 64k/32k budgets ([[reference/models-and-pricing]]). |
| `confidence` is arithmetic over `probabilities`, not a second learned estimate: TypeSafe's adapter uses `(p_max − 1/K)/(1 − 1/K)` | same, reading `confidence_metrics.py` | code read | `verified` for [[reference/system-one-adapter]]; `unverified` for the hosted API — [[concepts/confidence]] says the formula is unpublished. A sharp distribution can still be confidently wrong. |
| Explainer threads mostly restate launch numbers | — | restatement | Not evidence; idea lists at best. |

**One person's inference, not TypeSafe's description.** Archer Hume reconstructs Jev from those probes as a causal transformer ending in a readout instead of a decode loop: `state` held once as a shared prefix, each question an isolated suffix branch, the options read as one list before a single decision position, probably a sparse (MoE) backbone. TypeSafe has published only the parallel-output and RLCD claims ([[concepts/system-one]]); the rest is black-box deduction, as he says himself. A mental model, not a contract.

## Where it disappointed or broke

| Finding | Who | Evidence quality | Failure condition |
|---|---|---|---|
| Standalone browser control solved only **25/49** tasks; WebMCP tool menus took it to **49/49** | [@0xidanlevin](https://x.com/0xidanlevin/status/2100937437325205568) | measured, open repo (WindTunnel) | Jev must pick *the right next step*, not just a valid control. Author's framing: their harness, not a limit on Jev. |
| Cannot generate tool arguments; needed Mercury 2.5 beside it for search queries | same | measured | `verified` — jaggedness §9. Budget for a second small model. |
| Form-fill decision accuracy **83.6%** hosted vs **99.7%** for a task-trained local specialist | [@trycua](https://x.com/trycua/status/2101437979180904640) | measured, narrow | The specialist was fine-tuned for that exact convention. Decision-level, not end-to-end. |
| Hosted round trip **260–280 ms** vs **7–9 ms** local | same | measured, different boundaries | Network is the floor; no sub-100 ms loops. |
| Four overlapping Choice options ("high priority"/"urgent"/"needs review"/"flag") gave confident answers that were internally inconsistent "about a third of the time" | [@cyrilXBT](https://x.com/cyrilXBT/status/2101515130798297448) | anecdotal, self-reported (~6 h to fix) | `verified` as a schema bug, not a model bug — jaggedness §1, §7. Options must be mutually exclusive. |
| A 0.6B open replica beat Jev on ViZDoom Basic (**128/128** vs **56/128**), tied on Predict Position (11/128) | [TianyuCodings/NanoJev](https://github.com/TianyuCodings/NanoJev) | `unverified` — replica-run, Jev-side harness unchecked | Spatial/game control is not a System One strength; a task-trained tiny model can win. |
| **Too expensive for high-frequency control.** $0.042/Mtok is ~7x a DeepSeek V4.1 Flash cache read; 10 Hz at 10k context ≈ **$15/h** | [@jpschroeder](https://x.com/jpschroeder/status/2100952337904767258) | measured arithmetic | Price `verified` ([[reference/models-and-pricing]]); arithmetic checks out. He wants ~10x cheaper, on-device, for robotics. |
| Jev-scored compaction caveats, per the plugin's own README | [tamaratran/fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction) | author-stated | Full state re-sent **per request**, so long history costs a request per handful of questions; sizes are character estimates, not a tokenizer; "a probability is not a proof that a result is safe to delete"; falls back to the built-in summary on failure. |
| `/compact` timing needed a **private, hand-labelled eval set** (40 sessions) plus prompt hill-climbing | [@kunchenguid](https://x.com/kunchenguid/status/2101032677940117875) | anecdotal | Thresholds are yours to find; he shifts precision→recall as the window fills. |
| Per-minute BTC buy/hold/sell signal "performed poorly"; GPT-6 Astra did slightly better via news. Spelling an answer letter by letter was "rough" | Ryan Vogel via [@startupideaspod](https://x.com/startupideaspod/status/2101029801821941933) | anecdotal | `verified` direction — time series and generation are documented weaknesses ([[concepts/jaggedness-jev-1-13]]). He keeps Jev advisory, off portfolios. |
| Packet splitting: **7/8** packets exact vs **8/8** for GPT-5.6 Luna; classification tied at 40/40 | [jerryjliu/docjev](https://github.com/jerryjliu/docjev) | measured, open repo, 40 real PDFs | Boundary detection is the weaker half; gate low-confidence boundaries to review. |
| Keyword/regex rules vs Jev on tricky small sets: support routing 4/12 vs **11/12** (miss at 0.53 confidence, hits ≥0.75); changelog "major release" 1/8 vs 8/8 | [Nate B. Jones](https://unlock-ai.natebjones.com/guides/jev-shaped-problems) | measured by the guide's authors, tiny sets | Not production accuracy. The useful part is the confidence gap between hits and misses. |
| `Score` 1-10 on headline quality ordered items sensibly but rated the author's own published headlines 5-7 | same | measured | Rank with a score; not a pass mark. |
| Options `16GB` and `16 GB` offered separately split the probability; "is this laptop working?" missed a listing that mentioned a liquid-spill repair | same | measured | `verified` — overlapping options and literal reading ([[concepts/jaggedness-jev-1-13]]). Dedupe options; add the question that would have caught it. |
| **The option list itself moves probabilities.** Reversing a ticket's options shifted the top probability ~0.84–0.89 → 0.93–0.96 (a reference option read 16/16 last, 12/16 first, 11/16 middle, 48/48 from `state`); adding one irrelevant option shifted the log-odds between two existing ones +0.38 → +0.11, lower in all ten blocks | [Archer Hume](https://archerhume.com/posts/jevs-architecture-unmasked) | measured, ~330 requests | `unverified` — not in the docs, and it qualifies "give the full list, not a shortlist" ([[concepts/choice]]). A 0.9 gate can flip on a permutation; shuffle options in your eval and re-tune when the set changes. |
| **Do not assume determinism**: identical requests returned slightly different probabilities, duplicate questions in one request included; response key order varied | same | measured | `unverified` — docs promise "extremely consistent" ([[concepts/jaggedness-jev-1-13]] §8), not bit-exact. Don't key caches or tests on exact values. |
| `output_tokens` is a **billing figure, not generation**, and the tokenizer matches no public one, so local estimates drift | same | measured; figures in [[ideas/measurements]] | Never read decode speed or generation out of it; leave headroom under the 32k/64k caps. |
| AgentRun's own caveats: crash recovery does not cover model nodes or nested workflows yet; on one task the author matched every development label and scored no better on the reserved set | Grep.ai, [@MiguelriosEN](https://x.com/MiguelriosEN/status/2101029313906987422) | vendor self-report | Keep a held-out set the author never sees. |

No post in this corpus reports a *reproducible* correctness failure of Jev itself; the failures above are schema design, missing generation, harness scope, latency, price, and — new from the architecture probes — sensitivity to option order and option-set composition.

## What this means when advising on a project

- Re-measure on the project's own data (method: [[guides/testing-and-evaluation]]); every number here is n≈1.
- Expect the workflow multiplier, not the headline one: gains scale with the decision-to-generation ratio.
- Prefer narrow, mutually exclusive questions, and treat the option list as tuning surface: overlapping options produced confident nonsense, and order or an added option moved probabilities. Shuffle options in your eval; re-check thresholds when the set changes.
- Keep Jev out of anything that must generate (tool arguments, queries, summaries); pair it with a small fast LLM.
- No control loops faster than ~4 Hz or under a 250 ms budget (260–280 ms round trips); at high frequency, price it per hour, not per decision.
- Don't ship Jev-scored compaction without an eval set and a fallback; both published tools have one.
- Watch benchmark circularity: TypeSafe scores against an Astra+Fable average; replicas score against TypeSafe's records.
- Model routing may not pay for itself mid-session: Diogo Almeida (TypeSafe's CEO) works the arithmetic and finds routing to a cheaper model and back can cost *more*, because the large model re-processes the context — on his assumed mix, pure large-model was about two-thirds the cost ([post](https://x.com/CompleteSkeptic/status/2101894250401271876)). See [[ideas/patterns-agents]] P02.

## Related

- [[ideas/measurements]] — the numbers behind these verdicts; [[ideas/consult]] — the project walkthrough
- [[ideas/patterns]] — community patterns with fit verdicts; [[ideas/community-repos]] — the projects tested
- [[concepts/jaggedness-jev-1-13]] — the official failure-mode list these confirm; [[concepts/workflow-evals]] — where 193.6x / 444.6x comes from
- [[guides/testing-and-evaluation]] — running your own measurement

## Sources

Posts, repos and essays listed in this page's frontmatter (`raw/x/`, `raw/x-repos/`, `raw/community/`, `raw/nate/`); URLs inline above. The probe study is raw/community/archerhume-jevs-architecture-unmasked.md (https://archerhume.com/posts/jevs-architecture-unmasked), 2026-09-17.