$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Line-by-line search

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · search · choice · noul · thresholds

TL;DR Prefix every line with an id (L000…L217), make the ids the Choice options, and the ranking is the answer's probabilities. Because Choice probabilities always sum to 1, some line ranks first even when nothing answers the query — so send a Noul in the same request asking whether any line addresses it, and threshold that separately (>= 0.7 answered, < 0.35 absent, in between partial).

Goal

Build semantic search over GitHub's Terms of Service: return the lines that answer a plain-language question, and detect when the document has no answer. You end up with find(), which returns the exists probability and one relevance score per line.

Three parts:

  1. Tag each line with an ID so Jev can point to it.
  2. Use a Choice question to rank those line IDs by how well they answer the query.
  3. In the same request, use a Noul question to check whether the document contains an answer at all.

Inputs / state shape

The state is one big string: the document with every line prefixed by its id.

GIST = (
    "https://gist.githubusercontent.com/eugene-shvarts/900632789a24983d5678ffd508dd01f6"
    "/raw/cf9c2ab422d568deade949ef0a06bed6896964b9/github-tos.txt"
)

LINES = fetch_document(GIST).splitlines()   # 218 clauses


def line_id(i: int) -> str:
    return f"L{i:03d}"


DOCUMENT = "\n".join(f"{line_id(i)}| {line}" for i, line in enumerate(LINES))

DOCUMENT looks like this:

L052| You own Your Content. If you post Content you did not create, you are responsible for...
L053| You grant us and other Users the licenses in Sections D.4–D.8. These licenses apply...
L054| 4. License Grant to Us

218 lines, 43,980 characters. TYPESAFE_MODEL = "jev-1.12". The state stays unchanged between searches; only the questions carry the query.

Questions asked

Two questions per request, both parameterized by the query string.

where — a Choice over the 218 line ids:

def where_question(query: str) -> Choice:
    return Choice(
        instructions=f'Which line of the document contains the answer to: "{query}"?',
        criteria={line_id(i): None for i in range(len(LINES))},
    )

The option descriptions are None because the document already contains the text for each ID.

exists — a Noul with explicit criteria:

def exists_question(query: str) -> Noul:
    return Noul(
        instructions=f'Does any line of the document address or answer: "{query}"?',
        criteria=NoulCriteria(
            true="At least one line of the document states or directly implies the answer",
            false="No line of the document addresses this",
        ),
    )

Unlike the Choice probabilities, the Noul probability does not depend on the other options, so it can fall near zero when the document has no answer.

Combining logic in code

import os
import urllib.request

from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), timeout=120.0
)


def _find(model: str, state: str, where: Choice, exists: Noul) -> dict:
    response = client.system_one(
        state=state,
        questions={"where": where, "exists": exists},
        model=model,
    )
    probabilities = response.answers["where"].probabilities
    return {
        "exists": response.answers["exists"].noul,
        "relevance": [probabilities.get(line_id(i), 0.0) for i in range(len(LINES))],
    }


def find(query: str) -> dict:
    return _find(TYPESAFE_MODEL, DOCUMENT, where_question(query), exists_question(query))


FOUND, ABSENT = 0.7, 0.35  # present answers typically read >=0.9, absent <=0.05


def verdict(exists: float) -> str:
    if exists >= FOUND:
        return "answered in this document"
    return "not in this document" if exists < ABSENT else "partially addressed"


def show(query: str, top: int = 4) -> dict:
    result = find(query)
    print(f'"{query}"')
    print(f"  exists {result['exists']:.2f} -> {verdict(result['exists'])}")
    ranked = sorted(range(len(LINES)), key=lambda i: result["relevance"][i], reverse=True)
    for i in ranked[:top]:
        bar = "#" * max(1, round(result["relevance"][i] * 12))
        preview = LINES[i][:58].rstrip()
        print(f"  {line_id(i)}  {result['relevance'][i]:.2f}  {bar:<12}  {preview}")
    return result

The relevance list keeps one score per line, in document order; probabilities.get(line_id(i), 0.0) defaults any id the answer omits to zero. system_one answers both questions in one pass — the state is sent once, so adding the existence check costs only a small amount of extra output.

The cookbook is explicit that its thresholds are starting points: "These thresholds separate the examples below, but tune them against your own documents before using them in production."

Results / what the cookbook reports

Four queries: two with direct answers, one with no answer, one partial.

218 lines, 43,980 characters

"who owns the code I upload?"
  exists 0.98 -> answered in this document
  L052  0.95  ###########   You own Your Content. If you post Content you did not crea
  L046  0.02  #             Short version: You own content you create, but you allow u
  L051  0.02  #             3. Ownership and License Grants
  L217  0.01  #             Questions about the Terms of Service? Contact us through t

"can GitHub kick me off the platform without warning?"
  exists 0.97 -> answered in this document
  L168  0.97  ############  GitHub has the right to suspend or terminate your access t
  L167  0.03  #             3. GitHub May Terminate
  L000  0.00  #             Effective date: April 27, 2026 · A. Definitions
  L001  0.00  #             Short version: We use these basic terms throughout the agr

"do I have to take disputes to arbitration?"
  exists 0.14 -> not in this document
  L205  0.86  ##########    Except to the extent applicable law provides otherwise, th
  L168  0.02  #             GitHub has the right to suspend or terminate your access t

"can minors use GitHub with parental permission?"
  exists 0.46 -> partially addressed
  L029  0.90  ###########   You must be age 13 or older. While we are thrilled to see
  L012  0.07  #             "User," "You," and "Your" refer to the individual person,

The two failure-detection cases are the point:

The ranking tells you where to look; the exists score tells you whether the result answers the question.

Adapting it to a new domain

Swap the URL in fetch_document(); every other line of the script works off LINES. Then re-tune FOUND and ABSENT on your own documents and queries, and decide what "partially addressed" should route to.

Gotchas

Related

Sources