$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Re-ranking

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · retrieval · rerank · noul · bm25

TL;DR Fast search (BM25) narrows thousands of passages to a shortlist; re-ranking scores each query-candidate pair independently with one Noul and sorts by the returned probability. On 40 CLERC legal queries over 3,565 passages, 1,200 Jev calls moved top-1 accuracy from 5% to 18%, top-5 from 15% to 35%, top-10 from 38% to 62%, costing $0.0645.

Goal

Find the one document in thousands that answers a specific query, in two steps: cut the pile to a shortlist with a method fast enough to run on everything, then apply a more accurate step to that shortlist.

Why a Noul rather than an LLM score: a general model can produce scores, but you must invent a scoring scale and prompt the model to apply the same standard to every candidate, repeated calls can produce different scores for the same pair, and generation adds time and cost to a task that only needs one number. The question's criteria define what counts as true and false; Jev applies them to every pair and returns the noul directly — and that noul is the sort key.

Inputs / state shape

The state is a two-field dict, one query excerpt against one candidate passage:

state={"query_excerpt": query, "candidate_passage": candidate}

Dataset: CLERC, US federal court opinions, streamed from https://huggingface.co/datasets/jhu-clsp/CLERC/resolve/main/teva_train_dir/train_data.jsonl.gz. Each row has:

Constants: TYPESAFE_MODEL = "jev-1.12", PRICE = (0.042, 0.00) ($ per 1M input/output tokens, jev-1.12 as of 2026-08), N_ROWS = 170 CLERC rows pooled into the shared corpus, N_QUERIES = 40 rows evaluated, TOP_K = 30 candidates handed to the re-ranker per query. The pooled corpus is 3,565 passages; the first 20 pooled rows are held out and 40 queries are sampled from the rest, so the other 130 rows only ever appear as candidates.

Questions asked

One question, asked 1,200 times. Verbatim:

is_cited_source = Noul(
    instructions=(
        "The query excerpt comes from a US federal court opinion and was written "
        "immediately around a citation to a precedent; the citation itself has been "
        "removed. Could the candidate passage be from that cited precedent — does it "
        "establish the specific legal proposition the query excerpt invokes at its "
        "citation point?"
    ),
    criteria=NoulCriteria(
        true=(
            "The candidate passage states or establishes the specific rule, standard, "
            "holding, or fact pattern that the query excerpt attributes to its removed "
            "citation."
        ),
        false=(
            "The candidate passage is merely on a similar topic or doctrine; it does not "
            "supply the specific proposition the query excerpt relies on."
        ),
    ),
)

The false criterion is doing the real work: it names the near miss (same topic, wrong proposition) rather than describing an unrelated passage, which is what separates rank 1 from rank 5.

The cookbook also shows the minimal shape in pseudocode:

question = Noul(
    instructions="Is this candidate the cited case?",
    criteria=NoulCriteria(
        true="The candidate states the specific rule the query cites.",
        false="The candidate is only on a similar topic.",
    ),
)
response = client.system_one(state={...}, questions={"is_cited_source": question})
response.answers["is_cited_source"].noul  # -> 0.87

Combining logic in code

The whole re-ranker, in two lines of concept:

nouls = {candidate: ask_typesafe(query, candidate) for candidate in shortlist}
reranked = sorted(shortlist, key=lambda c: nouls[c], reverse=True)  # highest noul first

The real loop, trimmed of charting:

import json
import os
from concurrent.futures import ThreadPoolExecutor

import msgspec
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
TOP_K = 30

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)


def score_candidate(model: str, query: str, candidate: str, question_json: str) -> dict:
    """One TypeSafe call about one (query, candidate) pair: a noul, plus token usage."""
    # the SDK takes a question as its JSON dict, so the cached string decodes straight in
    question = json.loads(question_json)
    response = client.system_one(
        state={"query_excerpt": query, "candidate_passage": candidate},
        questions={"is_cited_source": question},
        model=model,
    )
    return {
        "noul": response.answers["is_cited_source"].noul,
        "input_tokens": response.usage.input_tokens or 0,
        "output_tokens": response.usage.output_tokens or 0,
    }


# 40 queries x 30 candidates = 1,200 independent calls — cheap enough to fire all at once.
pair_list = [(q, c) for q in queries for c in candidates[q]]
question_json = msgspec.json.encode(is_cited_source).decode()
with ThreadPoolExecutor(max_workers=12) as pool:
    results = pool.map(
        lambda p: score_candidate(TYPESAFE_MODEL, queries[p[0]], corpus[p[1]], question_json),
        pair_list,
    )

pair_scores = {q: {} for q in queries}
for (q, c), result in zip(pair_list, results):
    pair_scores[q][c] = result

reranked = {
    q: sorted(candidates[q], key=lambda c: -pair_scores[q][c]["noul"]) for q in queries
}

The fast-search step is BM25 and nothing else — keeping it simple leaves the attention on re-ranking, and the choice of fast search is a side issue because re-ranking only ever sees the passages that make the shortlist:

def bm25_rankings(corpus: dict[str, str], queries: dict[str, str], k: int = 100):
    """Rank every passage in the corpus by word overlap with each query."""
    import bm25s

    cids = list(corpus)
    retriever = bm25s.BM25()
    retriever.index(bm25s.tokenize([corpus[c] for c in cids], stopwords="en"))
    qids = list(queries)
    idxs, _ = retriever.retrieve(
        bm25s.tokenize([queries[q] for q in qids], stopwords="en"), k=min(k, len(cids))
    )
    return {q: [cids[i] for i in idxs[row]] for row, q in enumerate(qids)}


candidates = {q: ranked[:TOP_K] for q, ranked in bm25_rankings(corpus, queries).items()}

Results / what the cookbook reports

Fast search alone. The shortlist contains the correct passage for 100% of the 40 queries, but that passage is the top-ranked one only 5% of the time. Re-ranking cannot add a passage BM25 did not select, so with 100% shortlist recall it can focus purely on position.

After re-ranking:

threshold fast search + Jev re-rank
Top 1 5% 18%
Top 5 15% 35%
Top 10 38% 62%

Cost, printed verbatim:

1200 TypeSafe calls used 1,536,002 input and 25,200 output tokens, costing $0.0645.

Setup caveat stated by the cookbook: each CLERC row contains one correct passage and 20 negatives, but the walkthrough pools 170 rows into one shared corpus, so BM25 selects 30 candidates from the whole 3,565-passage corpus, not only that row's 20 negatives.

The cookbook also flags its own simplification: it asked one question per pair for clarity, and a real application would ask several questions about the same pair in one call — see Speculative fan-out and Cookbook: Parallel questions.

Adapting it to a new domain

  1. Keep your existing fast search (BM25, dense embeddings, or a hybrid); measure its shortlist recall first, because re-ranking's ceiling is whatever fraction of queries have the answer on the shortlist.
  2. Write one Noul whose true states the specific relation you want and whose false names the plausible near-miss.
  3. Send {query_field: ..., candidate_field: ...} as state, one call per pair, concurrently.
  4. Sort descending on answer.noul. Add more questions per pair in the same call when you need more than one relevance signal.

Gotchas

Related

Sources