---
title: "Cookbook: Re-ranking"
type: cookbook
tags: [cookbook, retrieval, rerank, noul, bm25]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/cookbooks__rerank_typesafe.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "BM25 builds a 30-candidate shortlist per CLERC query, then one Noul per query-candidate pair re-sorts it: top-1 rises 5% to 18%, top-10 38% to 62% for $0.0645."
---

# Cookbook: Re-ranking

> **TL;DR** Fast search (BM25) narrows thousands of passages to a shortlist; re-ranking scores each query-candidate pair independently with one `Noul` and sorts by the returned probability. On 40 CLERC legal queries over 3,565 passages, 1,200 Jev calls moved top-1 accuracy from **5% to 18%**, top-5 from **15% to 35%**, top-10 from **38% to 62%**, costing **$0.0645**.

## Goal

Find the one document in thousands that answers a specific query, in two steps: cut the pile to a shortlist with a method fast enough to run on everything, then apply a more accurate step to that shortlist.

Why a `Noul` rather than an LLM score: a general model can produce scores, but you must invent a scoring scale and prompt the model to apply the same standard to every candidate, repeated calls can produce different scores for the same pair, and generation adds time and cost to a task that only needs one number. The question's `criteria` define what counts as true and false; Jev applies them to every pair and returns the noul directly — and that noul **is** the sort key.

## Inputs / state shape

The state is a two-field dict, one query excerpt against one candidate passage:

```python
state={"query_excerpt": query, "candidate_passage": candidate}
```

Dataset: [CLERC](https://aclanthology.org/2025.findings-naacl.441/), US federal court opinions, streamed from `https://huggingface.co/datasets/jhu-clsp/CLERC/resolve/main/teva_train_dir/train_data.jsonl.gz`. Each row has:

- **Query** — an opinion excerpt with a citation removed.
- **Gold** — the passage the removed citation pointed to, the one correct answer.
- **Candidates** — every other passage in the pooled corpus.

Constants: `TYPESAFE_MODEL = "jev-1.12"`, `PRICE = (0.042, 0.00)` ($ per 1M input/output tokens, jev-1.12 as of 2026-08), `N_ROWS = 170` CLERC rows pooled into the shared corpus, `N_QUERIES = 40` rows evaluated, `TOP_K = 30` candidates handed to the re-ranker per query. The pooled corpus is **3,565 passages**; the first 20 pooled rows are held out and 40 queries are sampled from the rest, so the other 130 rows only ever appear as candidates.

## Questions asked

One question, asked 1,200 times. Verbatim:

```python
is_cited_source = Noul(
    instructions=(
        "The query excerpt comes from a US federal court opinion and was written "
        "immediately around a citation to a precedent; the citation itself has been "
        "removed. Could the candidate passage be from that cited precedent — does it "
        "establish the specific legal proposition the query excerpt invokes at its "
        "citation point?"
    ),
    criteria=NoulCriteria(
        true=(
            "The candidate passage states or establishes the specific rule, standard, "
            "holding, or fact pattern that the query excerpt attributes to its removed "
            "citation."
        ),
        false=(
            "The candidate passage is merely on a similar topic or doctrine; it does not "
            "supply the specific proposition the query excerpt relies on."
        ),
    ),
)
```

The `false` criterion is doing the real work: it names the *near miss* (same topic, wrong proposition) rather than describing an unrelated passage, which is what separates rank 1 from rank 5.

The cookbook also shows the minimal shape in pseudocode:

```python
question = Noul(
    instructions="Is this candidate the cited case?",
    criteria=NoulCriteria(
        true="The candidate states the specific rule the query cites.",
        false="The candidate is only on a similar topic.",
    ),
)
response = client.system_one(state={...}, questions={"is_cited_source": question})
response.answers["is_cited_source"].noul  # -> 0.87
```

## Combining logic in code

The whole re-ranker, in two lines of concept:

```python
nouls = {candidate: ask_typesafe(query, candidate) for candidate in shortlist}
reranked = sorted(shortlist, key=lambda c: nouls[c], reverse=True)  # highest noul first
```

The real loop, trimmed of charting:

```python
import json
import os
from concurrent.futures import ThreadPoolExecutor

import msgspec
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
TOP_K = 30

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)


def score_candidate(model: str, query: str, candidate: str, question_json: str) -> dict:
    """One TypeSafe call about one (query, candidate) pair: a noul, plus token usage."""
    # the SDK takes a question as its JSON dict, so the cached string decodes straight in
    question = json.loads(question_json)
    response = client.system_one(
        state={"query_excerpt": query, "candidate_passage": candidate},
        questions={"is_cited_source": question},
        model=model,
    )
    return {
        "noul": response.answers["is_cited_source"].noul,
        "input_tokens": response.usage.input_tokens or 0,
        "output_tokens": response.usage.output_tokens or 0,
    }


# 40 queries x 30 candidates = 1,200 independent calls — cheap enough to fire all at once.
pair_list = [(q, c) for q in queries for c in candidates[q]]
question_json = msgspec.json.encode(is_cited_source).decode()
with ThreadPoolExecutor(max_workers=12) as pool:
    results = pool.map(
        lambda p: score_candidate(TYPESAFE_MODEL, queries[p[0]], corpus[p[1]], question_json),
        pair_list,
    )

pair_scores = {q: {} for q in queries}
for (q, c), result in zip(pair_list, results):
    pair_scores[q][c] = result

reranked = {
    q: sorted(candidates[q], key=lambda c: -pair_scores[q][c]["noul"]) for q in queries
}
```

The fast-search step is BM25 and nothing else — keeping it simple leaves the attention on re-ranking, and the choice of fast search is a side issue because re-ranking only ever sees the passages that make the shortlist:

```python
def bm25_rankings(corpus: dict[str, str], queries: dict[str, str], k: int = 100):
    """Rank every passage in the corpus by word overlap with each query."""
    import bm25s

    cids = list(corpus)
    retriever = bm25s.BM25()
    retriever.index(bm25s.tokenize([corpus[c] for c in cids], stopwords="en"))
    qids = list(queries)
    idxs, _ = retriever.retrieve(
        bm25s.tokenize([queries[q] for q in qids], stopwords="en"), k=min(k, len(cids))
    )
    return {q: [cids[i] for i in idxs[row]] for row, q in enumerate(qids)}


candidates = {q: ranked[:TOP_K] for q, ranked in bm25_rankings(corpus, queries).items()}
```

## Results / what the cookbook reports

**Fast search alone.** The shortlist contains the correct passage for **100% of the 40 queries**, but that passage is the top-ranked one only **5%** of the time. Re-ranking cannot add a passage BM25 did not select, so with 100% shortlist recall it can focus purely on position.

**After re-ranking:**

| threshold | fast search | + Jev re-rank |
|---|---|---|
| Top 1 | 5% | 18% |
| Top 5 | 15% | 35% |
| Top 10 | 38% | 62% |

Cost, printed verbatim:

```
1200 TypeSafe calls used 1,536,002 input and 25,200 output tokens, costing $0.0645.
```

Setup caveat stated by the cookbook: each CLERC row contains one correct passage and 20 negatives, but the walkthrough pools 170 rows into one shared corpus, so BM25 selects 30 candidates from the whole 3,565-passage corpus, not only that row's 20 negatives.

The cookbook also flags its own simplification: it asked one question per pair for clarity, and a real application would ask several questions about the same pair in one call — see [[patterns/fan-out]] and [[cookbooks/parallel-questions]].

## Adapting it to a new domain

1. Keep your existing fast search (BM25, dense embeddings, or a hybrid); measure its shortlist recall first, because re-ranking's ceiling is whatever fraction of queries have the answer on the shortlist.
2. Write one `Noul` whose `true` states the specific relation you want and whose `false` names the plausible near-miss.
3. Send `{query_field: ..., candidate_field: ...}` as state, one call per pair, concurrently.
4. Sort descending on `answer.noul`. Add more questions per pair in the same call when you need more than one relevance signal.

## Gotchas

- **Re-ranking cannot rescue a bad shortlist.** 100% recall at `TOP_K = 30` here; if your fast search drops the gold, no amount of re-ranking helps.
- **Cost scales as queries × `TOP_K`.** 1,200 calls for 40 queries. Budget accordingly, and batch multiple questions per pair rather than multiple calls.
- **The `false` criterion is where accuracy comes from.** A vague false ("the candidate is unrelated") will not separate the same-doctrine near misses that dominate legal retrieval.
- **18% top-1 is still low in absolute terms** — this is a hard retrieval task and the cookbook reports it as a relative improvement, not a solved problem.
- **Install line.** `pip install bm25s datasets matplotlib "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a helper package from TypeSafe's private index, and `pypi.typesafe.ai` returned **404 publicly as of 2026-09-17** — install `pip install typesafe-sdk bm25s datasets` and reimplement `JsonCache(Path("json_cache.json"))`, a decorator that memoizes each call's return value into a JSON file keyed on its arguments so re-rendering replays the published numbers with no API spend.
- The published run used `jev-1.12`; this wiki documents `jev-1.13.0`.

## Related

- [[concepts/noul]] — how a yes/no question becomes a sort key
- [[patterns/fan-out]] — asking several questions about one pair in one call
- [[cookbooks/semantic-find]] — searching by meaning inside a single document
- [[cookbooks/classifying-rag-passages]] — deciding which retrieved passages reach the answering model
- [[cookbooks/parallel-questions]] — the batching measurement this cookbook defers to
- [[reference/models-and-pricing]] — the rate behind the $0.0645 figure
- [[cookbooks/overview]] — the full cookbook catalog

## Sources

- raw/docs/cookbooks__rerank_typesafe.md (https://docs.typesafe.ai/cookbooks/rerank_typesafe.md)