---
title: "Testing and evaluating a Jev workflow"
type: guide
tags: [testing, evaluation, calibration, consistency, thresholds]
created: 2026-09-17
updated: 2026-09-21
confidence: high
sources:
  - raw/nate/jev-shaped-problems.md
  - wiki/cookbooks/consistency-choice.md
  - wiki/cookbooks/consistency-noul.md
  - wiki/cookbooks/classification-using-confidence.md
  - wiki/concepts/workflow-evals.md
  - wiki/concepts/jaggedness-jev-1-13.md
  - wiki/concepts/confidence.md
  - wiki/entities/blog-antibenchmaxxing.md
  - wiki/reference/system-one-adapter.md
  - wiki/reference/legal-and-data.md
  - wiki/reference/models-and-pricing.md
  - wiki/guides/writing-instructions-and-criteria.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "Measure accuracy, calibration, repeatability, latency and cost on your own labelled set; pick thresholds from confidence buckets; regression-test jaggedness."
---

# Testing and evaluating a Jev workflow

> **TL;DR** Build a small labelled set, then measure five things: accuracy against your labels, calibration (accuracy per confidence bucket), repeatability across repeats of the same input, latency, and cost from `usage.input_tokens`. Pick thresholds from the calibration table, not from a cookbook. Regression-test the [[concepts/jaggedness-jev-1-13|jaggedness]] edge cases. And note before you publish: MCA §2.3(f) prohibits publishing benchmarks about the Services.

## What to measure

| Metric | How | Why it matters |
|---|---|---|
| **Accuracy** | Agreement between your workflow's final action and a label per case | The only metric that speaks to your application. Note TypeSafe's own "accuracy" means agreement with an ensemble reference, not truth ([[concepts/workflow-evals]]). |
| **Calibration** | Bucket answers by `confidence` (or by `noul`) and report accuracy per bucket | Calibration is a property of *groups*, never of one answer ([[concepts/confidence]]). This table is what sets your thresholds. |
| **Consistency / repeatability** | Run the same input N times; report per-question standard deviation and how often the decision flips | Jev is **not** deterministic ([[cookbooks/consistency-noul]]). |
| **Latency** | Wall clock around the call, measured from where your service runs | TypeSafe's published figures are measured "from our laptops on the West Coast" ([[concepts/workflow-evals]]). |
| **Cost** | `response.usage.input_tokens / 1e6 * 0.042` | Input tokens only; output tokens are free ([[reference/models-and-pricing]]). Price after the fact, so a rate change needs no new calls. |

Measure the **workflow's decision**, not just each answer. A question can wobble while the composed action stays put — and a rock-steady question can feed a decision that flips because the threshold sits exactly where the spread lands.

## Building a labelled set

1. **Collect real inputs**, including the boring majority and the genuinely hard cases. The consistency cookbooks each use a single input "built to sit on the fence"; a production set needs both kinds.
2. **Label the outcome you act on** — the queue, the route, the approve/hold/deny — not the intermediate probabilities.
3. **Split tuning from validation.** Tune wording and thresholds on one slice, validate on a held-out slice ([[guides/writing-instructions-and-criteria]]). Higher confidence on the tuning slice does not establish that an answer is right.
4. **If you have no labels**, TypeSafe's own move is an ensemble of expensive reasoning models answering every question in the harness — its workflow evals use the average of GPT-6 Astra and Claude Fable 5.1 at high thinking as the reference ([[concepts/workflow-evals]]). Carry the caveat with the numbers: a reference built from two competitors biases toward them, and the model being scored can never beat the reference by construction.
5. **Record the label's provenance.** The classification cookbook is explicit that its gold labels are self-reported SIC codes filtered so the filing text supports them, so the numbers "measure the recipe rather than the state of EDGAR's metadata" ([[cookbooks/classification-using-confidence]]).

Fifty to a few hundred cases is enough to start: the classification cookbook's headline result rests on 60 filings.

## Using confidence buckets to find the threshold

Score every case, sort by `confidence`, bucket, and report accuracy per bucket. The worked example: one `Choice` over 75 SIC groups, `confidence >= 0.9` on 30 of 60 filings was **27/30 (90%)** correct, while the 30 below were **12/30 (40%)** at group level and **70%** when reported one level up the taxonomy ([[cookbooks/classification-using-confidence]]).

Read three things off that table:

- **Where accuracy falls off** — that is your floor.
- **What the floor costs** — the share of traffic it sends to a fallback. In the choice-consistency cookbook a `0.60` top-probability floor bought 99.2% decision agreement while still acting automatically on 74.2% of answers, routing 25.8% to review ([[cookbooks/consistency-choice]]).
- **What the fallback should be.** A coarser label, a human, or a reasoning model — the cookbook's point is that an unconfident case comes back one level up instead of being dropped.

Two mechanical warnings. Gate a Noul on `noul` itself — Noul answers have no `confidence` field. And where a specific statistical rule applies, use `probabilities` rather than `confidence`; the choice-consistency cookbook states outright that its rule "uses the returned probabilities, not the API's separate `confidence` field."

## A 50-case check, then shadow mode (community method)

A short go/no-go procedure from [Nate B. Jones's member guide](https://unlock-ai.natebjones.com/guides/jev-shaped-problems) (community tier, used with his permission). It complements the bucket method above: use it first, when you have little labelled data.

1. Pull about 50 real past cases where you already know the right answer.
2. Run them through the Jev questions. Save the picked answer and its `confidence` for every case.
3. Look at every miss and write down its confidence. Compare with what your current process did on the same 50.
4. Set the threshold in the gap between the confidence of your hits and the confidence of your misses. In his 12-message support test the one miss came back at 0.53 and every hit at 0.75 or higher, so any gate between the two worked (his numbers, tiny set).
5. **No clean gap: this decision is not a fit yet.** Split the question into smaller ones, or leave it with the LLM.
6. **Misses at high confidence point at the question, not the threshold.** Check for two options that overlap (his example: `16GB` and `16 GB` offered separately split the probability) or one question that is really two ([[guides/writing-instructions-and-criteria]]).
7. Wire it in **shadow mode**: Jev runs on live traffic and logs input id, answers, confidences and what the existing process did, while the old path still makes the real decision. Compare after about a week; switch over only if the log holds up. Keep the old path as the low-confidence fallback ([[patterns/confidence-routing]]).

## Self-consistency tests

The two consistency cookbooks are the method, not just results. Both hold one input fixed, run the rubric **15 times**, and report per-question probability standard deviation, how often the top answer changes, and cost and latency per call.

| | [[cookbooks/consistency-noul]] | [[cookbooks/consistency-choice]] |
|---|---|---|
| Rubric | 14 Nouls (claims triage) | 8 Choices (moderation) |
| Repeats | 15, plus six LLM conditions | 15, plus six LLM conditions |
| Mean probability std dev | `0.0102` | `0.0098` |
| Per call | 111 ms, $0.000043 | 114 ms, $0.000046 |
| What moved | `covered` spanned 0.43–0.53, **crossing a 0.5 threshold** | 2 of 8 questions changed their top label |
| Mitigation | Band: `< 0.30` no, `0.30–0.70` uncertain → human, `> 0.70` yes | Abstain to `uncertain` below a `0.60` top probability |

Four harness mechanics worth copying verbatim:

- A **rubric fingerprint** — a digest of the state plus every question's text — in the cache key, so editing a question forces fresh samples instead of serving stale ones.
- A per-sample **`uid`** so each repeat is a distinct call. Both cookbooks acknowledge this is a confound: the `uid` is in the state, so the run measures response-to-a-changed-state and run-to-run variation together.
- Logging **`response.model`** on every call, because `jev-latest` is an alias that can move mid-run. All 15 calls returned `jev-1.13.0`.
- Reporting **repeatability separately from accuracy**. The choice cookbook says three times that repeatability is not accuracy; a competitor scored 100% agreement in the same run with no abstentions.

The actionable output is a list of questions whose spread straddles a decision threshold. Widen the band around them, re-word them, or decompose them — a question that abstains on *every* repeat is telling you the rubric is under-specified, not that the model failed.

## Comparing against an LLM

`system-one-adapter` answers the same `typesafe_sdk` questions with an OpenAI or Anthropic model and returns a `SystemOneResponse` subclass, so your state, questions, and composition code stay byte-identical and only the client swaps ([[reference/system-one-adapter]]):

```python
from system_one_adapter import SystemOneAdapterClient

client = SystemOneAdapterClient(
    structured_outputs=True,
    llm_answer_mode="probabilities",
    normalize_probabilities=True,
)
response = client.system_one(STATE, QUESTIONS, provider="anthropic", model="claude-haiku-4-5")
print(response.usage.latency, response.usage.input_tokens_total, response.usage.n_retries)
```

It adds `usage.latency`, `input_tokens_total` / `output_tokens_total` across retries, retry counts, and per-attempt traces in `response.debug["llm_attempts"]` — the raw material for a like-for-like cost, speed, and agreement comparison. Price the LLM side with that provider's rates; the cookbooks' `TYPESAFE_PRICE = (0.042, 0.00)` covers the Jev side only.

## Regression tests for jaggedness edge cases

Keep a standing suite of cases aimed at the documented failure modes ([[concepts/jaggedness-jev-1-13]]), run on every question edit and every model bump:

| Edge case | Test |
|---|---|
| Literal reading | An input that satisfies the letter of the instruction but not the intent — and vice versa. |
| Negation | The question and its negation on the same input. They are **not** complements: one documented pair summed to 1.19. Assert your code never relies on that. |
| Numbers | An input whose correct answer requires counting or arithmetic. The right result is that your **code** produced it, not a question. |
| Dates | Mixed formats, relative references ("last Thursday"), quarter and window boundaries. Assert the parts came from Choices and the comparison from `datetime`. |
| Adversarial content | State containing an injected instruction, a misleading framing, or text arguing for its own classification. Jev does not treat state as hostile by default. |
| Indirection | A question about a property of a property; confirm the flattened rewrite scores better. |
| Cross-type invariants | If any code compares a Noul to a Choice probability, or a threshold tuned on one type to the other, delete it. |

Also pin the model. Thresholds tuned against `jev-latest` shift silently when the alias moves; pin `jev-1.13.0` and re-run the suite before adopting a new version ([[reference/models-and-pricing]]).

## A minimal harness

Runnable with `pip install typesafe-sdk` (0.6.0) and `TYPESAFE_API_KEY` set. Replace `CASES` and `QUESTIONS` with your own.

```python
"""Minimal Jev evaluation harness: accuracy, calibration, repeatability, latency, cost."""

import statistics
from collections import Counter, defaultdict
from time import perf_counter

from typesafe_sdk import Choice, TypeSafeClient

MODEL = "jev-1.13.0"          # pin; jev-latest is an alias that moves
REPEATS = 15                  # matches the consistency cookbooks
PRICE_PER_MTOK_INPUT = 0.042  # output tokens are free
BUCKETS = [0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 1.01]

QUESTIONS = {
    "department": Choice(
        instructions="Which team should handle this ticket?",
        criteria={
            "billing": "Charges, invoices, refunds, subscriptions",
            "technical": "Bugs, outages, integrations",
            "other": "Anything else",
        },
    ),
}

CASES = [
    {"id": "t1", "state": "I was charged twice for order A-104.", "label": "billing"},
    {"id": "t2", "state": "The export button 500s on Safari.", "label": "technical"},
    # ... your held-out cases
]


def evaluate() -> None:
    hits: list[tuple[float, bool]] = []
    per_case_labels: dict[str, Counter] = defaultdict(Counter)
    per_case_conf: dict[str, list[float]] = defaultdict(list)
    latencies: list[float] = []
    cost = 0.0
    models = Counter()

    with TypeSafeClient(timeout=30.0) as client:
        for case in CASES:
            for repeat in range(REPEATS):
                started = perf_counter()
                response = client.system_one(
                    state={"uid": f"{case['id']}:{repeat}", "ticket": case["state"]},
                    questions=QUESTIONS,
                    model=MODEL,
                )
                latencies.append(perf_counter() - started)
                models[response.model] += 1
                cost += (response.usage.input_tokens or 0) / 1e6 * PRICE_PER_MTOK_INPUT

                answer = response.answers["department"]
                per_case_labels[case["id"]][answer.choice] += 1
                per_case_conf[case["id"]].append(answer.confidence)
                if repeat == 0:  # score accuracy on the first draw only
                    hits.append((answer.confidence, answer.choice == case["label"]))

    # 1. Accuracy
    print(f"accuracy {sum(ok for _, ok in hits)}/{len(hits)}")

    # 2. Calibration: accuracy per confidence bucket -> read your threshold off this
    for low, high in zip(BUCKETS, BUCKETS[1:]):
        band = [ok for conf, ok in hits if low <= conf < high]
        if band:
            print(f"  confidence [{low:.2f},{high:.2f})  n={len(band):3d}  acc={sum(band) / len(band):.2%}")

    # 3. Repeatability: label flips and confidence spread per case
    for case_id, counts in per_case_labels.items():
        spread = statistics.pstdev(per_case_conf[case_id])
        flag = "  <-- FLIPS" if len(counts) > 1 else ""
        print(f"  {case_id}: {dict(counts)} conf_std={spread:.4f}{flag}")

    # 4/5. Latency and cost
    print(f"median latency {statistics.median(latencies) * 1000:.0f} ms over {len(latencies)} calls")
    print(f"total cost ${cost:.6f}   models answered: {dict(models)}")


if __name__ == "__main__":
    evaluate()
```

Cache results to a file keyed on case id, repeat index, rubric fingerprint, and model, so re-analysis costs nothing — that is what the cookbooks' `JsonCache` does ([[cookbooks/overview]]).

## Before you publish the numbers

TypeSafe's public position is "run your own private evals, treat public ones with a grain of salt" ([[entities/blog-antibenchmaxxing]]), and the company commits to no standard benchmark table, dated snapshots retired rather than hill-climbed, and publishing its own unflattering evidence.

The contract points the other way. **MCA §2.3(f) prohibits customers from publishing "benchmarks or performance information about the Services"** ([[reference/legal-and-data]]). Internal evaluation is exactly what the docs ask for; publishing your Jev numbers is a contractual matter — get permission first. This is a summary, not legal advice.

## Related

- [[ideas/field-reports]] — what independent testers measured and where Jev disappointed (community tier)
- [[concepts/confidence]] — what `confidence` is and why calibration is a group property
- [[concepts/jaggedness-jev-1-13]] — the failure modes the regression suite targets
- [[concepts/workflow-evals]] — TypeSafe's own harness and its caveats
- [[cookbooks/consistency-noul]], [[cookbooks/consistency-choice]] — the repeatability method in full
- [[cookbooks/classification-using-confidence]] — the confidence-bucket table worked end to end
- [[reference/system-one-adapter]] — same questions, an LLM behind them
- [[guides/writing-instructions-and-criteria]] — the iteration loop a failing case feeds
- [[guides/agent-integration-playbook]] — where testing sits in the build
- [[entities/blog-antibenchmaxxing]] — TypeSafe on benchmarks
- [[reference/legal-and-data]] — MCA §2.3(f)

## Sources

- raw/nate/jev-shaped-problems.md (https://unlock-ai.natebjones.com/guides/jev-shaped-problems) — the 50-case check and shadow-mode procedure; community tier, with the author's permission
- wiki/cookbooks/consistency-choice.md, wiki/cookbooks/consistency-noul.md, wiki/cookbooks/classification-using-confidence.md, wiki/cookbooks/overview.md
- wiki/concepts/workflow-evals.md, wiki/concepts/jaggedness-jev-1-13.md, wiki/concepts/confidence.md
- wiki/entities/blog-antibenchmaxxing.md
- wiki/reference/system-one-adapter.md, wiki/reference/legal-and-data.md, wiki/reference/models-and-pricing.md, wiki/reference/python-sdk.md
- wiki/guides/writing-instructions-and-criteria.md