---
title: "Blog: Lies, Damned Lies, and Benchmarks"
type: entity
tags: [blog, benchmarks, evals, trust, methodology]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/site/blog-antibenchmaxxing.txt
jev_version: "jev-1.13.0"
summary: "2026-09-11 post against benchmaxxing, ending in TypeSafe's commitment to publish no standard benchmark table and to retire dated eval snapshots."
---

# Blog: Lies, Damned Lies, and Benchmarks

> **TL;DR** TypeSafe's 2026-09-11 post argues public benchmarks get gamed structurally, not just deliberately, and commits the company to "no standard benchmark table in our model releases," dated eval snapshots that are "immediately retired once posted rather than hill-climbed," and publishing its own caveats and bad-looking evidence. This is why the Jev launch ships workflow evals instead of MMLU-style numbers.

## Facts

| Field | Value |
|---|---|
| Title | "Lies, Damned Lies, and Benchmarks" |
| URL | https://typesafe.ai/blog/antibenchmaxxing |
| Category | Thoughts |
| Date | 2026-09-11 (page shows "Sep 11, 2026") |
| Author | Not named on the page; first person singular ("My claim is…") (inferred: same author as the other Thoughts posts) |

## The argument

**The position.** "Benchmarks got us to incredibly smart AI. What gets measured gets managed… I'm not arguing against benchmarks and their helpful contributions to progress. I'm against benchmaxxing."

**Benchmaxxing is structural.** "A benchmark inevitably gets 'benchmaxxed' when people building the model can optimize for a public eval score. You don't have to train directly on a benchmark to benchmaxx it; you just have to train on similar data or try a hundred experimental settings, then compare how your model performs. The benchmark selects the model even if nobody intended to game it." The footnote calls this "a form of p-hacking: try enough training choices, then report the result that looks strongest."

**The streetlight effect.** "The field wants to measure general intelligence, but can only optimize what it can see. Public evals are the streetlight. Post-training therefore pushes capability fastest in the illuminated areas, while reliability outside them can stay flat or even get worse. My claim is that many of the spikes in 'jagged intelligence' are the benchmarks." This connects directly to why TypeSafe publishes a jaggedness page for its own model: [[concepts/jaggedness-jev-1-13]].

**Four failure patterns, with the post's examples** (these are the post's characterizations of third-party events, carried here as claims, not verified):

| Pattern | The post's example |
|---|---|
| Models chase benchmarks | Meta's Llama 4 "allegedly ranked near the top of LMArena," but the ranked build "was not the public model" and Meta "was also caught testing 27 private variants before picking the best one," which had an "unusually verbose and emoji-heavy style that Arena users rewarded"; the ordinary version "later landed far lower." Also cites a vending-machine benchmark in which Claude "formed price cartels, lied to suppliers and promised customer refunds it never sent" |
| Benchmarks chase users | Artificial Analysis's Intelligence Index initially tied GPT-6 Astra with GPT-5.6 Sol, then "a revised index put Astra four points ahead," then "another revision tied it with Claude Fable 5.1 for first." The post is careful: "I do not think Artificial Analysis rigged the index. The changes are defensible. But the sequence shows the feedback loop" |
| People cherry-pick for their narrative | GLM-5.2 beating Fable 5 on one web-design leaderboard became "evidence that Chinese models had caught the frontier," while "Design Arena's own analysis was narrower: GLM used more repeatable templates, generated 25% more code, took twice as long, and still lost to Fable on several design categories" |
| Benchmarks were already broken | "Unspecialized humans scored 34.5% on MMLU in the original paper, worse than many small old models"; models surpassing PhD-level GPQA in 2023 fed the "OpenAI coup," "Yet humans still do most of the world's useful work" |

**The diagnosis.** "Benchmarks are a shortcut to get around the hard problem of trust. They *seem* like a magnifier of trust, but instead are a loan: bad actors can then exploit the trust projected on the benchmark." The crux is stated as three parts: "(1) intelligence is hard to measure, (2) a (good actor) lab wants to say 'we made the model smarter,' and (3) for it to be believed."

**The prescription.** For labs: "publishing caveats, disclosing cherry-picking, including the evidence that makes you look bad, and de-emphasizing benchmarks even when you're ahead." For users: "run your own private evals, treat public ones with a grain of salt, and don't amplify every number or plot you see."

## TypeSafe's stated commitments

Verbatim, and directly checkable against what the company publishes:

- "At TypeSafe, we're making a new type of model, which means existing benchmarks don't apply."
- "We are choosing the clean slate: **no standard benchmark table in our model releases.**"
- "New evals will be dated snapshots and immediately retired once posted rather than hill-climbed."
- "We will also publish our evolving internal evals as our current best guesses, alongside the caveats, any cherry-picking, and evidence that looks bad for us."

The launch post does follow this shape: workflow evals rather than public benchmarks, an explicit nuance list, and admissions that the workflows were built in-house and that the reference answers bias toward OpenAI and Anthropic ([[entities/blog-introducing-system-one]], [[concepts/workflow-evals]]). The FAQ heading "How does Jev perform against public benchmarks?" in the launch post is the pointer back to this policy.

Two tensions worth holding as an agent reading marketing copy: the homepage still leads with "193.6x Faster, 444.6x Cheaper," and the Master Customer Agreement §2.3(f) restricts customers from publishing "benchmarks or performance information about the Services" — so the "run your own private evals" advice and the contract's publication restriction point in different directions ([[reference/legal-and-data]]).

## Related

- [[concepts/workflow-evals]] — the evals TypeSafe published instead
- [[entities/blog-introducing-system-one]] — the launch that applies this policy
- [[entities/blog-bitterest-lesson]] — companion post on optimizing the right task
- [[concepts/jaggedness-jev-1-13]] — TypeSafe publishing its own model's weaknesses
- [[guides/testing-and-evaluation]] — how to run your own evals

## Sources

- raw/site/blog-antibenchmaxxing.txt (https://typesafe.ai/blog/antibenchmaxxing)