Cookbook: SDE cascade

TL;DR Structured-data extraction as a two-rung cascade: gpt-5.4-mini extracts, a per-field Noul battery on Jev scores P(wrong) for seven failure modes, and if any field flag exceeds FIRE_T = 0.7 the record is re-extracted by gpt-5.5 at reasoning_effort="high". Schema validation catches structural errors and never semantic ones; the verifier is what catches a schema-valid fabrication.

Goal

Big reasoning models extract structured data well but are slow and expensive; small models are cheap but make mistakes. A cascade gets most of the quality at a fraction of the cost.

Models and prices ($ per 1M tokens, input / output; standard rates checked September 15, 2026):

The algorithm: Extract cheap → Verify with a per-field yes/no Noul battery, each returning P(something is wrong) → Escalate to the expensive model if a verifier signal fires; otherwise keep the cheap answer.

Both extraction rungs use text-mode OpenAI, not structured outputs, tool calls, or JSON mode, because a schema-following mistake is not the mistake an LLM is expected to make, and when one does fail to follow the schema it is almost always very confused, so constrained decoding does not fix the underlying issue.

Inputs / state shape

Row 516 of scrapegraphai/scrapegraphai-100k at revision 4bb9fba1dff9181c5acdb60a5a26fea62fa54fe9. It is an NYU events-calendar page ("Fall 2024 Census Date") whose scrape captured only calendar nav and boilerplate — there is no registration date and no description on the page.

• prompt: Find registration open date fall semester for New York University in New York, NY for the 2024-2025 school year.
• schema: two required string properties, registration_open_date (description: "The date that registration opens for the fall semester. MUST be in the format mm/dd/yyyy. For example, for a college in the 2024-2025 school year, it might be something like 09/05/2024. Return a blank string if you are unsure.") and description (description: "A brief description of the registration open date. For example, 'Registration opens for the fall semester'."). Note the schema's description field ships an example value in its own field description.

The verifier's state is a five-key dict:

state = {
    "system_message": EXTRACT_SYSTEM,
    "instruction": "Extract the structured record from this document",
    "source_text": row["content"],
    "schema": schema,
    "extraction": record,
}

The mini extraction that the rest of the walkthrough explains (hard-coded because gpt-5.4-mini is very stochastic on this input and invents a different description on nearly every run, even at temperature=0):

mini_record = {
    "registration_open_date": "",
    "description": "Registration opens for the fall semester",
}

jsonschema.Draft202012Validator(schema).is_valid(mini_record) prints True. It is schema-valid and still wrong: the blank date matches the page, but the description is fabricated — mini parrots the schema's own example.

Questions asked

Every question is a Noul framed so that true = something is wrong (escalate).

Per-field battery (MAIN_QUESTIONS, metric → (question, NoulCriteria)), verbatim:

Empty-field question — fields that are None, "" or an empty collection get only this one:

ABSENCE_QUESTION = (
    "The `extracted_field` is empty, null, or an empty collection. Does the source text contain the "
    "information the `field_spec` describes, making the empty result wrong?"
)
ABSENCE_CRITERIA = NoulCriteria(
    true="a value was wrongly omitted", false="returning nothing is correct"
)

Whole-record head — computed and displayed to contrast a holistic judgment with the per-field heads, but the gate does not use it:

OVERALL_JUDGE = (
    "Is this extracted record an incorrect extraction -- some value unsupported by the source or "
    "not conforming to the schema, required information missing or wrong, or some field hallucinated -- "
    "so it should be escalated to a smarter model?"
)
OVERALL_JUDGE_CRITERIA = NoulCriteria(
    true="the record is an incorrect extraction",
    false="the record is a correct extraction",
)

The key structural trick: the question text goes into the state-like instructions dict, not a bare string:

questions[f"{name}::{metric}"] = Noul(
    instructions={
        "field_spec": spec,
        "extracted_field": value,
        "main_question": question,
    },
    criteria=criteria,
)

where spec = {"path": name, "type": typ, "description": ..., "required": bool}. Question ids are field::metric, plus the single __overall__::judge. The full pipeline also has a spurious head for whole containers and an overall difficulty score, not shown in the walkthrough.

Combining logic in code

import json
import os

import jsonschema
from openai import OpenAI
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

MINI = "gpt-5.4-mini"        # rung 0: cheap + fast
REASONING = "gpt-5.5"        # rung 1: strong, run with reasoning_effort="high"
TS_MODEL = "jev-1.12"        # the TypeSafe verifier model
FIRE_T = 0.7                 # escalate if any per-field P(wrong) exceeds this

oai = OpenAI()
ts = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=30.0)

EXTRACT_SYSTEM = (
    "You extract structured data from documents. Return only values supported by the text. "
    "Follow any value format specified by the schema or its field descriptions."
)


def build_questions(record: dict) -> dict[str, Noul]:
    """One holistic ``__overall__::judge`` head plus a per-field battery, keyed ``field::metric``."""
    questions: dict[str, Noul] = {
        "__overall__::judge": Noul(instructions=OVERALL_JUDGE, criteria=OVERALL_JUDGE_CRITERIA),
    }
    for name, value in record.items():
        spec = field_spec(name)
        if is_empty(value):
            questions[f"{name}::absence_wrong"] = Noul(
                instructions={"field_spec": spec, "extracted_field": value,
                              "main_question": ABSENCE_QUESTION},
                criteria=ABSENCE_CRITERIA,
            )
            continue
        for metric, (question, criteria) in MAIN_QUESTIONS.items():
            if metric == "type_mismatch" and spec["type"] == "unknown":
                continue
            questions[f"{name}::{metric}"] = Noul(
                instructions={"field_spec": spec, "extracted_field": value,
                              "main_question": question},
                criteria=criteria,
            )
    return questions


def verify(record: dict) -> dict[str, float]:
    """Run the whole Noul battery over a record in one TypeSafe call; return {field::metric: P(true)}."""
    state = {
        "system_message": EXTRACT_SYSTEM,
        "instruction": "Extract the structured record from this document",
        "source_text": row["content"],
        "schema": schema,
        "extraction": record,
    }
    answers = ts.system_one(state=state, questions=build_questions(record), model=TS_MODEL).answers
    return {qid: ans.noul for qid, ans in answers.items()}


# The gate: escalate if ANY per-field flag fires. max-style, not a mean.
checks = verify(mini_record)
fired = {qid: p for qid, p in checks.items()
         if not qid.startswith("__overall__") and p > FIRE_T}
escalate = bool(fired)

final_record = (
    extract(REASONING, prompt, schema, content, reasoning_effort="high")
    if escalate
    else mini_record
)

The extractor parses the reply as-is; if json.loads fails it returns {}, the record-level analog of NaN, so every field reads as absent, the verifier flags it and the gate escalates — the safe direction.

Results / what the cookbook reports

Verifier output on the mini extraction (P(wrong), sorted):

qid                                      P(wrong)
--------------------------------------------------
description::hallucinated                    0.95  <== FIRES
description::off_target                      0.85  <== FIRES
description::unreasonable                    0.58
__overall__::judge                           0.56
description::incomplete                      0.16
registration_open_date::absence_wrong        0.14
description::format_violation                0.10
description::name_desc_mismatch              0.08
description::type_mismatch                   0.02

The signal concentrates on the field that is actually wrong: high on the fabricated description, low on the correctly-blank date, medium on a field that looks off without being clearly wrong. The holistic __overall__::judge head reads only 0.56 on the same record — which is the cookbook's argument for per-field heads over a blunt "is this whole thing good?" judge.

Gate and escalation:

any_flag gate (threshold 0.7): ESCALATE
  fired: description::hallucinated  (P=0.95)
  fired: description::off_target  (P=0.85)

mini      : {"registration_open_date": "", "description": "Registration opens for the fall semester"}
reasoning : {"description": "", "registration_open_date": ""}

field-level diff (mini -> final):
  description: 'Registration opens for the fall semester'  ->  ''

At scale (internal TypeSafe results, 100 scrapegraphai prompts). Same extract → verify → escalate loop, gpt-5.4-mini → gpt-5.5-reasoning, any_flag gate over the per-field heads; the gate threshold is swept 0→1 and every config plotted in (cost, quality) space. The four single models appear as black diamonds — the strongest, gpt-5.5-reasoning, sits at ≈0.81 quality for ≈$0.10/extraction. The cascade's pareto frontier sits up-and-left of every single model. TypeSafe notes the chart is a historical snapshot whose costs have not been recalculated at the current Jev rate.

Adapting it to a new domain

Replace the schema and the source text; field_spec() reads path, type, description and required straight out of JSON Schema (unwrapping anyOf/null for optional fields), so the battery generalizes without new question text. Tune FIRE_T against labeled examples, and sweep it to pick your own point on the cost/quality frontier.

Appendix A of the cookbook — what makes a good verifier signal:

• Narrow and grounded. One checkable yes/no about one field against the source, not a vague "is this extraction good?" Vague questions give mushy, uncalibrated scores.
• Bad = TRUE, with explicit criteria. Frame each question so the escalate case is the true case, and state what true/false mean.
• Per-field, then aggregate with max. A per-field flag localizes the error and stays sparse and strong; max ensures one confident red flag escalates instead of being averaged into silence.
• Independent and cheap. A dedicated verifier judging the output catches the extractor's own blind spots, and it has to be cheap or there are no savings left to capture.
• Separating / calibrated. High on real errors, low on correct ones, so a single threshold cleanly splits accept vs escalate. That separation is what pushes the pareto curve up-and-left.

Gotchas

• Schema validation is necessary but not sufficient. The fabricated record validates cleanly. "It catches structural errors, never semantic ones. That gap is the whole point."
• The holistic head is a demo, not the gate. Wiring __overall__::judge into any_flag would have under-fired here (0.56 < 0.7).
• Do not average field flags. A mean over nine heads would have buried a 0.95.
• Empty fields get one question, not seven — otherwise six of the seven heads are meaningless on "".
• type_mismatch is skipped when the spec type is unknown, so unparseable schemas do not generate a garbage flag.
• gpt-5.4-mini is stochastic on this input even at temperature=0; the walkthrough hard-codes one canonical fabrication. A real pipeline calls extract(MINI, prompt, schema, content, temperature=0) directly.
• Install line. pip install openai datasets jsonschema ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/. cooksafe is a helper package served from TypeSafe's package index, and pypi.typesafe.ai returned 404 publicly as of 2026-09-17 — install pip install typesafe-sdk openai datasets jsonschema and reimplement JsonCache(Path("json_cache.json")) (a decorator memoizing each call's return value into a JSON file keyed on its arguments; delete the file to re-run live) and make_playground_link(state, questions) (packs state and questions into a https://console.typesafe.ai/playground#share/... URL).

Related

• Noul (yes/no) questions — P(true) per hazard head
• Confidence vs probability — why calibration is what makes one threshold work
• Confidence-gated routing — the escalate/accept gate as a general pattern
• Speculative fan-out — the whole battery in one request
• Cookbook: Double-checking citations — the same "verify a model's output against the source" shape
• Cookbook: Guardrails for LLMs — another Noul battery plus thresholds you own
• Cookbooks overview — the full cookbook catalog

Sources

• raw/docs/cookbooks__sde_cascade.md (https://docs.typesafe.ai/cookbooks/sde_cascade.md)