--- title: "Cookbook: Structure recovery (autoformat)" type: cookbook tags: [cookbook, autoformat, noul, choice, markdown] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__autoformat.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "Rebuild Markdown from de-formatted plain text in two requests: Nouls stitch hard-wrapped lines, then Choices classify each block; code renders every character from the input." --- # Cookbook: Structure recovery (autoformat) > **TL;DR** Two sequential requests per document. **Pass 1**: one `Noul` per adjacent line pair — "does this line pick up mid-sentence?" — merged in code with a punctuation-dependent threshold (0.2 after a dangling line, 0.5 after terminal punctuation). **Pass 2**: per merged block, a `Choice` over six content types plus three companion questions (heading level, step-order Noul, callout kind) asked up front and read only when relevant. Code renders the Markdown, so no word changes. Memo run: 2 round trips, 10,211 tokens, 0.8s. ## Goal Take plain text whose markup was stripped — lines hard-wrapped mid-sentence, no heading markers, no bullets — and reconstruct Markdown headings, paragraphs, lists, quotes, code and callouts. The reason not to use a text generator: "A text-generation model could rewrite the text into Markdown, but a rewrite can also change the words. Here the model never generates text: it answers narrow questions about the document... and code does the rendering, so every character of the output comes from the input, and every judgment carries a probability." A companion rule the cookbook states explicitly: **direct evidence stays in code.** "Blank lines and explicit markers (`- `, `1.`, `#`) are read in code, never sent to the model to reconsider... The model gets only the questions code cannot answer from the text." ## Inputs / state shape The test document is a build-system migration memo fetched from a pinned gist (28 non-blank lines). Lines are normalized, blank-line gaps recorded, and every line is tagged with a short id that the model reads as ordinary text: ```python def to_lines(text: str) -> list[dict]: lines, gap = [], False for raw in text.split("\n"): stripped = re.sub(r"[\t ]+", " ", raw).strip() if not stripped: gap = bool(lines) # a leading blank is not a break continue lines.append({"text": stripped, "gap": gap}) gap = False return lines def tag(items: list[dict], prefix: str) -> str: return "\n".join( f"{chr(10) if item['gap'] else ''}{prefix}{i:03d}| {item['text']}" for i, item in enumerate(items) ) def line_id(i: int) -> str: return f"L{i:03d}" def block_id(i: int) -> str: return f"B{i:03d}" ``` The state for both passes is a single **tagged string** — `tag(LINES, "L")` in pass 1, `tag(blocks, "B")` in pass 2 — and questions refer to lines/blocks by those ids. The cookbook notes this is "the same scheme as the [semantic search cookbook]" ([[cookbooks/semantic-find]]). What the model sees: ``` L013| The cutover touches three teams, so check whether you are on this L014| list before you plan anything for Monday: L015| The platform team L016| The web client team L017| Whoever still owns the release tooling ``` ## Questions asked ### Pass 1 — stitch (one Noul per adjacent non-gapped pair) ```python def join_question(i: int) -> Noul: return Noul( instructions=f"Does line {line_id(i)} pick up mid-sentence, continuing a sentence left unfinished at the end of line {line_id(i - 1)}?", criteria=NoulCriteria( true="The line starts in the middle of a sentence that began on the previous line - the line break tore the sentence apart", false="The line begins a new sentence, item, heading, or thought of its own", ), ) ``` The rejected alternative wording, kept in the cookbook as a controlled comparison: ```python def naive_join_question(i: int) -> Noul: return Noul( instructions=f"Are lines {line_id(i - 1)} and {line_id(i)} part of the same paragraph?", criteria=NoulCriteria( true="The two lines belong to the same paragraph of running text", false="The two lines belong to different paragraphs or different pieces of content", ), ) ``` ### Pass 2 — classify (the whole specification) "These three dicts, plus the step question's true/false criteria inside `classify_questions` below, are the entire specification of the classifier. There is no other logic. To adapt the pipeline to your own documents, edit these descriptions." ```python TYPE_CRITERIA = { "heading": "A short label or title that names the document or the section that follows it - not a full sentence of content", "paragraph": "Running prose: one or more complete sentences of explanatory or narrative text", "list_item": "One entry in a list of parallel items - an ingredient, a feature, a task, an attendee; reads as one of several sibling entries", "quote": "Words attributed to a person or source - quoted speech, a citation, an excerpt someone else wrote", "code": "Computer code, a shell command, terminal output, or a config snippet meant to be read verbatim", "callout": "A warning, tip, or important note that interrupts the flow to flag something the reader must not miss", } HLEVEL_CRITERIA = { "title": "The title of the whole document", "section": "A major section heading within the document", "subsection": "A minor heading nested under a section", } CALLOUT_CRITERIA = { "note": "Neutral extra information the reader should be aware of", "tip": "A helpful suggestion or shortcut that makes things easier", "warning": "A caution about something that can go wrong or cause harm", } HEADING_MAX_CHARS = 90 # longer blocks can't render as headings, so don't ask def classify_questions(texts: list[str]) -> dict: questions = {} for i, text in enumerate(texts): bid = block_id(i) questions[f"type_{bid}"] = Choice( instructions=f"What kind of content is block {bid}?", criteria=TYPE_CRITERIA ) if len(text) <= HEADING_MAX_CHARS: questions[f"hlevel_{bid}"] = Choice( instructions=f"As a heading, what level would block {bid} occupy in this document's structure?", criteria=HLEVEL_CRITERIA, ) questions[f"step_{bid}"] = Noul( instructions=f"Is block {bid} an instruction in a sequence where the order of the items matters?", criteria=NoulCriteria( true="It is one step of a procedure - the items around it must happen in order", false="Order is irrelevant - it is a loose collection, or not a list item at all", ), ) questions[f"callout_{bid}"] = Choice( instructions=f"What kind of aside is block {bid}?", criteria=CALLOUT_CRITERIA ) return questions ``` Why companion questions are asked speculatively: "The types are not known yet, and waiting for them would mean a third round trip, so the companion questions are asked up front in the same request. Most of these answers are never read: the step probability of a paragraph means nothing and is simply ignored. An extra question adds little, since the state is most of the tokens and is sent once either way, while an extra round trip adds a full request of latency." See [[patterns/fan-out]]. ## Combining logic in code Setup and pass 1: ```python import os, re, urllib.request from pathlib import Path from time import perf_counter from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient TYPESAFE_MODEL = "jev-1.12" PRICE = (0.042, 0.00) # $ per 1M tokens (input, output); TypeSafe jev-1.12 as of 2026-09 client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=120.0) def stitch() -> dict: questions = {line_id(i): join_question(i) for i in range(1, len(LINES)) if not LINES[i]["gap"]} started = perf_counter() response = client.system_one( state=tag(LINES, "L"), questions=questions, model=TYPESAFE_MODEL ) return { "joins": [ response.answers[line_id(i)].noul if line_id(i) in response.answers else 0.0 for i in range(len(LINES)) ], "seconds": round(perf_counter() - started, 2), "usage": [response.usage.input_tokens, response.usage.output_tokens], } ``` The merge, with the punctuation-dependent cutoff: ```python JOIN_AFTER_DANGLING, JOIN_AFTER_TERMINAL = 0.2, 0.5 def ends_terminal(text: str) -> bool: return re.search(r'[.!?:;…]["\')\]]*$', text) is not None def merge(joins: list[float]) -> list[dict]: blocks = [] for i, line in enumerate(LINES): bar = (JOIN_AFTER_TERMINAL if i and ends_terminal(LINES[i - 1]["text"]) else JOIN_AFTER_DANGLING) if blocks and not line["gap"] and joins[i] >= bar: blocks[-1]["text"] += " " + line["text"] blocks[-1]["lines"].append(i) else: blocks.append({"text": line["text"], "lines": [i], "gap": line["gap"]}) return blocks ``` Pass 2 and rendering: ```python def classify(texts: list[str], gaps: list[bool]) -> dict: tagged = tag([{"text": t, "gap": g} for t, g in zip(texts, gaps)], "B") questions = classify_questions(texts) response = client.system_one(state=tagged, questions=questions, model=TYPESAFE_MODEL) judgments = [] for i in range(len(texts)): bid = block_id(i) type_answer = response.answers[f"type_{bid}"] hlevel = response.answers.get(f"hlevel_{bid}") judgments.append({ "type": type_answer.choice, "confidence": type_answer.confidence, "probabilities": type_answer.probabilities, "hlevel": hlevel.choice if hlevel else "section", "step": response.answers[f"step_{bid}"].noul, "callout": response.answers[f"callout_{bid}"].choice, }) return {"judgments": judgments, "n_questions": len(questions), "usage": [response.usage.input_tokens, response.usage.output_tokens]} STEP_THRESHOLD = 0.5 HEADING_MARK = {"title": "#", "section": "##", "subsection": "###"} CALLOUT_MARK = {"note": "NOTE", "tip": "TIP", "warning": "WARNING"} def to_markdown(blocks: list[dict]) -> str: groups = [] for b in blocks: if b["type"] in ("list_item", "code") and groups and groups[-1][0] == b["type"]: groups[-1][1].append(b) else: groups.append((b["type"], [b])) parts = [] for kind, items in groups: if kind == "list_item": ordered = sum(b["step"] for b in items) / len(items) >= STEP_THRESHOLD parts.append("\n".join( f"{n + 1}. {b['text']}" if ordered else f"- {b['text']}" for n, b in enumerate(items) )) elif kind == "code": parts.append("```\n" + "\n".join(b["text"] for b in items) + "\n```") elif kind == "heading": parts.append(f"{HEADING_MARK[items[0]['hlevel']]} {items[0]['text']}") elif kind == "quote": parts.append(f"> {items[0]['text']}") elif kind == "callout": parts.append(f"> [!{CALLOUT_MARK[items[0]['callout']]}]\n> {items[0]['text']}") else: parts.append(items[0]["text"]) return "\n\n".join(parts) + "\n" ``` Ordered-vs-bulleted is a **group** decision: the mean of the items' step probabilities against `STEP_THRESHOLD = 0.5`. "That threshold is a group-level decision no single question asked directly." ## Results the cookbook reports Model `jev-1.12`. Pass 1: 16 pair questions, one request, 0.32s → `28 lines -> 17 blocks (11 line breaks healed)`. Pass 2: 62 questions about 17 blocks, one request, 0.51s. Per-block classification (abridged from the cookbook's table): | block | type | conf | companion used | |---|---|---|---| | B000 | heading | 0.99 | level=title | | B002 | heading | 0.75 | level=section | | B004 | code | 1.00 | – | | B006 | paragraph | 0.43 | – | | B007–B009 (team names) | list_item | 0.99 / 1.00 / 0.99 | step=0.15 / 0.16 / 0.12 | | B011–B013 (to-dos) | list_item | 0.98 / 0.99 / 0.92 | step=0.86 / 0.87 / 0.90 | | B014 | callout | 0.65 | kind=warning | | B015 | quote | 0.99 | – | The step probabilities do the work: the three to-dos near 0.9 render as a numbered list, the three team names near 0.1 as bullets, and the unmarked warning became `> [!WARNING]`. "Every word above is from the input. The pipeline only chose boundaries, types, and markup." ### Where the join thresholds come from Per-line join probabilities landed "in two separate bands: line breaks that split a sentence score 0.39 and up, breaks the author meant score close to zero." But the cutoff depends on the previous line's punctuation, "a fact code can read directly": true continuations scored as low as 0.39 (`make the switch for real.`), so a flat 0.5 would break paragraphs; meanwhile `L015| The platform team` follows a colon and scores 0.22, which would clear a flat 0.2 and swallow the list into its introducing sentence. "No single threshold works for both cases; once code checks the punctuation first, the two bands separate." ### Why "mid-sentence" and not "same paragraph" Same document, same request shape, only the wording changed: ``` mid-sentence same paragraph L015 The platform team 0.22 0.77 L016 The web client team 0.11 0.81 L017 Whoever still owns the release tooli 0.12 0.78 L020 Delete the old build cache directory 0.08 0.88 L021 Run the doctor script and fix anythi 0.05 0.91 blocks after merge: 17 (mid-sentence) vs 12 (same paragraph) ``` The lesson, quoted: "When a judgment call feeds a threshold, the question should name the narrowest fact that decides it. Here the wording is the difference between 17 blocks and 12." See [[guides/writing-instructions-and-criteria]]. ### The lowest-confidence block ``` "The cutover touches three teams, so check whether you are on this list before you plan anything for Monday:" confidence 0.43: paragraph 0.53, list_item 0.24, callout 0.19 ``` The cookbook's suggested UI treatment: "underline for review any block whose type confidence (the probability behind the winning choice) is under 0.55." ### Cost and latency ``` pass 1 16 questions 0.32s pass 2 62 questions 0.51s total 10,211 tokens 0.8s $0.0003 ``` **Source inconsistency:** the code output prints `$0.0003`, while the prose twice states "\$0.0015" for the same run. `$0.0003` is what `PRICE = (0.042, 0.00)` and 10,211 input tokens actually produce, so the prose figure appears to be stale (inferred). ## Adapting it to a new domain - "To adapt the pipeline to your own documents, edit these descriptions" — `TYPE_CRITERIA`, `HLEVEL_CRITERIA`, `CALLOUT_CRITERIA`, and the step Noul's `NoulCriteria`. Everything else is plumbing. - Add a block type by adding a `TYPE_CRITERIA` entry plus a branch in `to_markdown()`, and a companion question if it needs a sub-decision (inferred). - Re-derive the two join thresholds on your own text; they follow from where your model's probability bands fall, not from anything universal. - If your input kept its markers (`- `, `#`, `1.`), read them in code and skip those questions — that is the cookbook's stated principle. - `HEADING_MAX_CHARS = 90` suppresses a pointless question for long blocks; adjust to your rendering rules. ## Gotchas - **`cooksafe` is not publicly installable.** Install line: `pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper on a private index, and `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk` and reimplement: `JsonCache(Path("json_cache.json"))` is a decorator memoizing JSON-serializable return values to a file keyed by the call arguments (used here even on `fetch_document`, so the gist is only fetched once); `make_playground_link(state, questions, models=[...])` builds a `console.typesafe.ai/playground#share/...` URL. - **This cookbook requires a real key.** Unlike most others it uses `os.environ["TYPESAFE_API_KEY"]` (KeyError if unset) rather than a `"cache-only"` default. - **`NoulCriteria` is a distinct import** (`from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient`), with `true=` / `false=` fields. See [[reference/python-sdk-questions]]. - **Question wording is the single biggest lever here** — 17 blocks vs 12 from one reworded question, at identical cost. - **Pass 2 cannot be merged into pass 1.** "The blocks only exist once pass 1 has answered, so this is a second request." - **Speculative companion questions are cheap but not free**; they are worth it because the state dominates the token count. Check that trade-off if your blocks are short and numerous (inferred). - **Answers missing from the response are floored to 0.0** in `stitch()` (`if line_id(i) in response.answers else 0.0`) — gapped pairs are never asked, and that default encodes "do not merge". - **Model pinning.** All numbers are `jev-1.12`; `jev-latest` now resolves to `jev-1.13.0`, so the join thresholds should be re-derived. ## Related - [[cookbooks/overview]] — the cookbook index - [[cookbooks/semantic-find]] — the same line-id tagging scheme - [[concepts/noul]] — `Noul` and `NoulCriteria` - [[concepts/choice]] — `Choice` options and `confidence` - [[patterns/fan-out]] — many questions in one request - [[guides/writing-instructions-and-criteria]] — the "narrowest fact" rule - [[reference/models-and-pricing]] — the `$0.042` / MTok input price used here ## Sources - raw/docs/cookbooks__autoformat.md (https://docs.typesafe.ai/cookbooks/autoformat)