---
title: "Cookbook: SDE cascade"
type: cookbook
tags: [cookbook, extraction, verification, noul, cascade]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/cookbooks__sde_cascade.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "Extract with a cheap model, verify every field with a battery of Jev Nouls framed so true means wrong, and escalate to a reasoning model only when a flag fires."
---

# Cookbook: SDE cascade

> **TL;DR** Structured-data extraction as a two-rung cascade: `gpt-5.4-mini` extracts, a per-field `Noul` battery on Jev scores P(wrong) for seven failure modes, and if **any** field flag exceeds `FIRE_T = 0.7` the record is re-extracted by `gpt-5.5` at `reasoning_effort="high"`. Schema validation catches structural errors and never semantic ones; the verifier is what catches a schema-valid fabrication.

## Goal

Big reasoning models extract structured data well but are slow and expensive; small models are cheap but make mistakes. A cascade gets most of the quality at a fraction of the cost.

Models and prices (\$ per 1M tokens, input / output; standard rates checked **September 15, 2026**):

| rung | model | price |
|---|---|---|
| rung 0 (mini) | `gpt-5.4-mini` | $0.75 / $4.50 |
| rung 1 (reasoning) | `gpt-5.5` | $5.00 / $30.00 (roughly 7x the mini) |
| verifier | TypeSafe `jev-1.12` | $0.042 / $0.00 (output tokens are free) |

The algorithm: **Extract** cheap → **Verify** with a per-field yes/no Noul battery, each returning P(something is wrong) → **Escalate** to the expensive model if a verifier signal fires; otherwise keep the cheap answer.

Both extraction rungs use **text-mode OpenAI**, not structured outputs, tool calls, or JSON mode, because a schema-following mistake is not the mistake an LLM is expected to make, and when one does fail to follow the schema it is almost always very confused, so constrained decoding does not fix the underlying issue.

## Inputs / state shape

Row 516 of `scrapegraphai/scrapegraphai-100k` at revision `4bb9fba1dff9181c5acdb60a5a26fea62fa54fe9`. It is an NYU events-calendar page ("Fall 2024 Census Date") whose scrape captured only calendar nav and boilerplate — **there is no registration date and no description on the page**.

- `prompt`: `Find registration open date fall semester for New York University in New York, NY for the 2024-2025 school year.`
- `schema`: two required string properties, `registration_open_date` (description: *"The date that registration opens for the fall semester. MUST be in the format mm/dd/yyyy. For example, for a college in the 2024-2025 school year, it might be something like 09/05/2024. Return a blank string if you are unsure."*) and `description` (description: *"A brief description of the registration open date. For example, 'Registration opens for the fall semester'."*). Note the schema's `description` field ships an example value in its own field description.

The verifier's state is a five-key dict:

```python
state = {
    "system_message": EXTRACT_SYSTEM,
    "instruction": "Extract the structured record from this document",
    "source_text": row["content"],
    "schema": schema,
    "extraction": record,
}
```

The mini extraction that the rest of the walkthrough explains (hard-coded because `gpt-5.4-mini` is very stochastic on this input and invents a different `description` on nearly every run, even at `temperature=0`):

```python
mini_record = {
    "registration_open_date": "",
    "description": "Registration opens for the fall semester",
}
```

`jsonschema.Draft202012Validator(schema).is_valid(mini_record)` prints `True`. It is schema-valid and still wrong: the blank date matches the page, but the `description` is fabricated — mini parrots the schema's own example.

## Questions asked

Every question is a `Noul` framed so that **`true` = something is wrong (escalate)**.

**Per-field battery** (`MAIN_QUESTIONS`, metric → (question, `NoulCriteria`)), verbatim:

| metric | `main_question` | `criteria.true` | `criteria.false` |
|---|---|---|---|
| `name_desc_mismatch` | ``Does the `extracted_field` fail to match the field at `path` or the `description` in the `field_spec`? If the `description` is empty, judge against the `path` alone.`` | ``the `extracted_field` does not match the field name or its `description` `` | ``the `extracted_field` matches the field name and `description` `` |
| `type_mismatch` | ``Does the `extracted_field` violate the `type` declared in the `field_spec`?`` | ``the `extracted_field` violates the declared `type` `` | ``the `extracted_field` conforms to the declared `type` `` |
| `unreasonable` | ``Is the `extracted_field` one that a reasonable person would not have extracted for this `field_spec`?`` | `a reasonable person would not have extracted this value` | `the extraction is reasonable` |
| `hallucinated` | ``Is the `extracted_field` unsupported by, or absent from, the source text?`` | ``the `extracted_field` is a hallucination -- not supported by, or absent from, the source text`` | ``the `extracted_field` is supported by the source text`` |
| `off_target` | ``Does the source text fail to genuinely report the thing the `field_spec` describes, so the value was pulled from incidental text?`` | `the source does not genuinely provide this field -- the value was pulled from incidental text` | `the source genuinely reports this field` |
| `incomplete` | ``Does the `extracted_field` fail to capture a value the source supports (note whether the `field_spec` is `required`)?`` | `the field is wrongly empty, null, or missing a value the source supports` | `the field captures the value the source supports` |
| `format_violation` | ``Does the `extracted_field` violate the format or constraints implied by the `description`, the schema `type`, and the extraction instructions (e.g. date format, units, enum membership)?`` | ``the `extracted_field` violates the implied format or constraints`` | ``the `extracted_field` satisfies the format and constraints`` |

**Empty-field question** — fields that are `None`, `""` or an empty collection get only this one:

```python
ABSENCE_QUESTION = (
    "The `extracted_field` is empty, null, or an empty collection. Does the source text contain the "
    "information the `field_spec` describes, making the empty result wrong?"
)
ABSENCE_CRITERIA = NoulCriteria(
    true="a value was wrongly omitted", false="returning nothing is correct"
)
```

**Whole-record head** — computed and displayed to contrast a holistic judgment with the per-field heads, but **the gate does not use it**:

```python
OVERALL_JUDGE = (
    "Is this extracted record an incorrect extraction -- some value unsupported by the source or "
    "not conforming to the schema, required information missing or wrong, or some field hallucinated -- "
    "so it should be escalated to a smarter model?"
)
OVERALL_JUDGE_CRITERIA = NoulCriteria(
    true="the record is an incorrect extraction",
    false="the record is a correct extraction",
)
```

The key structural trick: the *question text* goes into the state-like `instructions` **dict**, not a bare string:

```python
questions[f"{name}::{metric}"] = Noul(
    instructions={
        "field_spec": spec,
        "extracted_field": value,
        "main_question": question,
    },
    criteria=criteria,
)
```

where `spec = {"path": name, "type": typ, "description": ..., "required": bool}`. Question ids are `field::metric`, plus the single `__overall__::judge`. The full pipeline also has a `spurious` head for whole containers and an overall `difficulty` score, not shown in the walkthrough.

## Combining logic in code

```python
import json
import os

import jsonschema
from openai import OpenAI
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

MINI = "gpt-5.4-mini"        # rung 0: cheap + fast
REASONING = "gpt-5.5"        # rung 1: strong, run with reasoning_effort="high"
TS_MODEL = "jev-1.12"        # the TypeSafe verifier model
FIRE_T = 0.7                 # escalate if any per-field P(wrong) exceeds this

oai = OpenAI()
ts = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=30.0)

EXTRACT_SYSTEM = (
    "You extract structured data from documents. Return only values supported by the text. "
    "Follow any value format specified by the schema or its field descriptions."
)


def build_questions(record: dict) -> dict[str, Noul]:
    """One holistic ``__overall__::judge`` head plus a per-field battery, keyed ``field::metric``."""
    questions: dict[str, Noul] = {
        "__overall__::judge": Noul(instructions=OVERALL_JUDGE, criteria=OVERALL_JUDGE_CRITERIA),
    }
    for name, value in record.items():
        spec = field_spec(name)
        if is_empty(value):
            questions[f"{name}::absence_wrong"] = Noul(
                instructions={"field_spec": spec, "extracted_field": value,
                              "main_question": ABSENCE_QUESTION},
                criteria=ABSENCE_CRITERIA,
            )
            continue
        for metric, (question, criteria) in MAIN_QUESTIONS.items():
            if metric == "type_mismatch" and spec["type"] == "unknown":
                continue
            questions[f"{name}::{metric}"] = Noul(
                instructions={"field_spec": spec, "extracted_field": value,
                              "main_question": question},
                criteria=criteria,
            )
    return questions


def verify(record: dict) -> dict[str, float]:
    """Run the whole Noul battery over a record in one TypeSafe call; return {field::metric: P(true)}."""
    state = {
        "system_message": EXTRACT_SYSTEM,
        "instruction": "Extract the structured record from this document",
        "source_text": row["content"],
        "schema": schema,
        "extraction": record,
    }
    answers = ts.system_one(state=state, questions=build_questions(record), model=TS_MODEL).answers
    return {qid: ans.noul for qid, ans in answers.items()}


# The gate: escalate if ANY per-field flag fires. max-style, not a mean.
checks = verify(mini_record)
fired = {qid: p for qid, p in checks.items()
         if not qid.startswith("__overall__") and p > FIRE_T}
escalate = bool(fired)

final_record = (
    extract(REASONING, prompt, schema, content, reasoning_effort="high")
    if escalate
    else mini_record
)
```

The extractor parses the reply as-is; if `json.loads` fails it returns `{}`, the record-level analog of NaN, so every field reads as absent, the verifier flags it and the gate escalates — the safe direction.

## Results / what the cookbook reports

Verifier output on the mini extraction (`P(wrong)`, sorted):

```
qid                                      P(wrong)
--------------------------------------------------
description::hallucinated                    0.95  <== FIRES
description::off_target                      0.85  <== FIRES
description::unreasonable                    0.58
__overall__::judge                           0.56
description::incomplete                      0.16
registration_open_date::absence_wrong        0.14
description::format_violation                0.10
description::name_desc_mismatch              0.08
description::type_mismatch                   0.02
```

The signal concentrates on the field that is actually wrong: high on the fabricated `description`, low on the correctly-blank date, medium on a field that looks off without being clearly wrong. The holistic `__overall__::judge` head reads only 0.56 on the same record — which is the cookbook's argument for per-field heads over a blunt "is this whole thing good?" judge.

Gate and escalation:

```
any_flag gate (threshold 0.7): ESCALATE
  fired: description::hallucinated  (P=0.95)
  fired: description::off_target  (P=0.85)
```

```
mini      : {"registration_open_date": "", "description": "Registration opens for the fall semester"}
reasoning : {"description": "", "registration_open_date": ""}

field-level diff (mini -> final):
  description: 'Registration opens for the fall semester'  ->  ''
```

**At scale (internal TypeSafe results, 100 scrapegraphai prompts).** Same `extract → verify → escalate` loop, `gpt-5.4-mini → gpt-5.5-reasoning`, `any_flag` gate over the per-field heads; the gate threshold is swept 0→1 and every config plotted in (cost, quality) space. The four single models appear as black diamonds — the strongest, `gpt-5.5-reasoning`, sits at **≈0.81 quality for ≈$0.10/extraction**. The cascade's pareto frontier sits **up-and-left of every single model**. TypeSafe notes the chart is a historical snapshot whose costs have not been recalculated at the current Jev rate.

## Adapting it to a new domain

Replace the schema and the source text; `field_spec()` reads `path`, `type`, `description` and `required` straight out of JSON Schema (unwrapping `anyOf`/`null` for optional fields), so the battery generalizes without new question text. Tune `FIRE_T` against labeled examples, and sweep it to pick your own point on the cost/quality frontier.

**Appendix A of the cookbook — what makes a good verifier signal:**

- **Narrow and grounded.** One checkable yes/no about one field against the source, not a vague "is this extraction good?" Vague questions give mushy, uncalibrated scores.
- **Bad = TRUE, with explicit criteria.** Frame each question so the *escalate* case is the `true` case, and state what `true`/`false` mean.
- **Per-field, then aggregate with `max`.** A per-field flag localizes the error and stays sparse and strong; `max` ensures one confident red flag escalates instead of being averaged into silence.
- **Independent and cheap.** A dedicated verifier judging the output catches the extractor's own blind spots, and it has to be cheap or there are no savings left to capture.
- **Separating / calibrated.** High on real errors, low on correct ones, so a single threshold cleanly splits accept vs escalate. That separation is what pushes the pareto curve up-and-left.

## Gotchas

- **Schema validation is necessary but not sufficient.** The fabricated record validates cleanly. "It catches structural errors, never semantic ones. That gap is the whole point."
- **The holistic head is a demo, not the gate.** Wiring `__overall__::judge` into `any_flag` would have under-fired here (0.56 < 0.7).
- **Do not average field flags.** A mean over nine heads would have buried a 0.95.
- **Empty fields get one question, not seven** — otherwise six of the seven heads are meaningless on `""`.
- **`type_mismatch` is skipped when the spec type is `unknown`**, so unparseable schemas do not generate a garbage flag.
- **`gpt-5.4-mini` is stochastic on this input** even at `temperature=0`; the walkthrough hard-codes one canonical fabrication. A real pipeline calls `extract(MINI, prompt, schema, content, temperature=0)` directly.
- **Install line.** `pip install openai datasets jsonschema ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a helper package served from TypeSafe's package index, and `pypi.typesafe.ai` returned **404 publicly as of 2026-09-17** — install `pip install typesafe-sdk openai datasets jsonschema` and reimplement `JsonCache(Path("json_cache.json"))` (a decorator memoizing each call's return value into a JSON file keyed on its arguments; delete the file to re-run live) and `make_playground_link(state, questions)` (packs state and questions into a `https://console.typesafe.ai/playground#share/...` URL).

## Related

- [[concepts/noul]] — P(true) per hazard head
- [[concepts/confidence]] — why calibration is what makes one threshold work
- [[patterns/confidence-routing]] — the escalate/accept gate as a general pattern
- [[patterns/fan-out]] — the whole battery in one request
- [[cookbooks/citation-check]] — the same "verify a model's output against the source" shape
- [[cookbooks/llm-guardrails]] — another `Noul` battery plus thresholds you own
- [[cookbooks/overview]] — the full cookbook catalog

## Sources

- raw/docs/cookbooks__sde_cascade.md (https://docs.typesafe.ai/cookbooks/sde_cascade.md)