--- title: "Cookbook: Line-by-line search" type: cookbook tags: [cookbook, search, choice, noul, thresholds] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__semantic_find.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "Tag a document's lines with ids, rank them with one Choice over those ids, and pair it with a Noul that says whether the document answers the query at all." --- # Cookbook: Line-by-line search > **TL;DR** Prefix every line with an id (`L000`…`L217`), make the ids the `Choice` options, and the ranking is the answer's `probabilities`. Because Choice probabilities always sum to 1, *some* line ranks first even when nothing answers the query — so send a `Noul` in the same request asking whether any line addresses it, and threshold that separately (`>= 0.7` answered, `< 0.35` absent, in between partial). ## Goal Build semantic search over GitHub's Terms of Service: return the lines that answer a plain-language question, and detect when the document has no answer. You end up with `find()`, which returns the `exists` probability and one relevance score per line. Three parts: 1. Tag each line with an ID so Jev can point to it. 2. Use a `Choice` question to rank those line IDs by how well they answer the query. 3. In the same request, use a `Noul` question to check whether the document contains an answer at all. ## Inputs / state shape The state is one big string: the document with every line prefixed by its id. ```python GIST = ( "https://gist.githubusercontent.com/eugene-shvarts/900632789a24983d5678ffd508dd01f6" "/raw/cf9c2ab422d568deade949ef0a06bed6896964b9/github-tos.txt" ) LINES = fetch_document(GIST).splitlines() # 218 clauses def line_id(i: int) -> str: return f"L{i:03d}" DOCUMENT = "\n".join(f"{line_id(i)}| {line}" for i, line in enumerate(LINES)) ``` `DOCUMENT` looks like this: ``` L052| You own Your Content. If you post Content you did not create, you are responsible for... L053| You grant us and other Users the licenses in Sections D.4–D.8. These licenses apply... L054| 4. License Grant to Us ``` **218 lines, 43,980 characters.** `TYPESAFE_MODEL = "jev-1.12"`. The state stays unchanged between searches; only the questions carry the query. ## Questions asked Two questions per request, both parameterized by the query string. **`where` — a `Choice` over the 218 line ids:** ```python def where_question(query: str) -> Choice: return Choice( instructions=f'Which line of the document contains the answer to: "{query}"?', criteria={line_id(i): None for i in range(len(LINES))}, ) ``` The option descriptions are `None` because the document already contains the text for each ID. **`exists` — a `Noul` with explicit criteria:** ```python def exists_question(query: str) -> Noul: return Noul( instructions=f'Does any line of the document address or answer: "{query}"?', criteria=NoulCriteria( true="At least one line of the document states or directly implies the answer", false="No line of the document addresses this", ), ) ``` Unlike the Choice probabilities, the Noul probability does not depend on the other options, so it can fall near zero when the document has no answer. ## Combining logic in code ```python import os import urllib.request from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient TYPESAFE_MODEL = "jev-1.12" client = TypeSafeClient( api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), timeout=120.0 ) def _find(model: str, state: str, where: Choice, exists: Noul) -> dict: response = client.system_one( state=state, questions={"where": where, "exists": exists}, model=model, ) probabilities = response.answers["where"].probabilities return { "exists": response.answers["exists"].noul, "relevance": [probabilities.get(line_id(i), 0.0) for i in range(len(LINES))], } def find(query: str) -> dict: return _find(TYPESAFE_MODEL, DOCUMENT, where_question(query), exists_question(query)) FOUND, ABSENT = 0.7, 0.35 # present answers typically read >=0.9, absent <=0.05 def verdict(exists: float) -> str: if exists >= FOUND: return "answered in this document" return "not in this document" if exists < ABSENT else "partially addressed" def show(query: str, top: int = 4) -> dict: result = find(query) print(f'"{query}"') print(f" exists {result['exists']:.2f} -> {verdict(result['exists'])}") ranked = sorted(range(len(LINES)), key=lambda i: result["relevance"][i], reverse=True) for i in ranked[:top]: bar = "#" * max(1, round(result["relevance"][i] * 12)) preview = LINES[i][:58].rstrip() print(f" {line_id(i)} {result['relevance'][i]:.2f} {bar:<12} {preview}") return result ``` The `relevance` list keeps one score per line, in document order; `probabilities.get(line_id(i), 0.0)` defaults any id the answer omits to zero. `system_one` answers both questions in one pass — the state is sent once, so adding the existence check costs only a small amount of extra output. The cookbook is explicit that its thresholds are starting points: "These thresholds separate the examples below, but tune them against your own documents before using them in production." ## Results / what the cookbook reports Four queries: two with direct answers, one with no answer, one partial. ``` 218 lines, 43,980 characters "who owns the code I upload?" exists 0.98 -> answered in this document L052 0.95 ########### You own Your Content. If you post Content you did not crea L046 0.02 # Short version: You own content you create, but you allow u L051 0.02 # 3. Ownership and License Grants L217 0.01 # Questions about the Terms of Service? Contact us through t "can GitHub kick me off the platform without warning?" exists 0.97 -> answered in this document L168 0.97 ############ GitHub has the right to suspend or terminate your access t L167 0.03 # 3. GitHub May Terminate L000 0.00 # Effective date: April 27, 2026 · A. Definitions L001 0.00 # Short version: We use these basic terms throughout the agr "do I have to take disputes to arbitration?" exists 0.14 -> not in this document L205 0.86 ########## Except to the extent applicable law provides otherwise, th L168 0.02 # GitHub has the right to suspend or terminate your access t "can minors use GitHub with parental permission?" exists 0.46 -> partially addressed L029 0.90 ########### You must be age 13 or older. While we are thrilled to see L012 0.07 # "User," "You," and "Your" refer to the individual person, ``` The two failure-detection cases are the point: - **Arbitration:** the ranking gives the closest line `0.86`, but `exists` is only `0.14`. The answer is not in the document. - **Parental permission:** the age rule ranks first at `0.90`, but it does not answer whether parental permission changes the rule — `exists 0.46` lands in the partial band. The ranking tells you where to look; the `exists` score tells you whether the result answers the question. ## Adapting it to a new domain Swap the URL in `fetch_document()`; every other line of the script works off `LINES`. Then re-tune `FOUND` and `ABSENT` on your own documents and queries, and decide what "partially addressed" should route to. ## Gotchas - **255-option ceiling.** A `Choice` question accepts up to 255 options, so this recipe searches documents of up to 255 lines in one request. Past that, search in two passes: one Choice picks a window of lines, and a second ranks the lines inside it. (218 lines here leaves little headroom.) - **Choice probabilities always sum to 1.** The top-ranked line is never evidence that an answer exists — that is exactly what `exists` is for. Never threshold on the relevance score alone. - **Line granularity is your choice of split.** The ToS is split into 218 *clauses*, not raw text lines; the quality of the split bounds the quality of the result (inferred). - **The two signals can disagree, and that is informative** — high relevance with low `exists` means "closest line, wrong question". - **Install line.** `pip install "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a helper package from TypeSafe's private index and `pypi.typesafe.ai` returned **404 publicly as of 2026-09-17** — install `pip install typesafe-sdk` and reimplement `JsonCache(Path("json_cache.json"))`, the decorator that memoizes each decorated call's return value into a JSON file keyed on its arguments. Here it wraps both `fetch_document` (so the gist is downloaded once) and `_find` (so the published answers replay without an API key or spend); delete the file and set `TYPESAFE_API_KEY` to run live. - Published run used `jev-1.12`; this wiki documents `jev-1.13.0`. ## Related - [[concepts/choice]] — probabilities over a fixed option set, and the 255-option limit - [[concepts/noul]] — the independent existence check - [[patterns/fan-out]] — both questions in one request over one state - [[cookbooks/rerank]] — searching across documents instead of within one - [[cookbooks/pre-parsed-value-extraction]] — the same "options are spans from the document" trick - [[cookbooks/citation-check]] — checking a quote against its source document - [[cookbooks/overview]] — the full cookbook catalog ## Sources - raw/docs/cookbooks__semantic_find.md (https://docs.typesafe.ai/cookbooks/semantic_find.md)