---
title: "Cookbook: Double-checking citations"
type: cookbook
tags: [cookbook, citation-check, choice, confidence, verification]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/cookbooks__citation_check.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "One Choice question decides whether a quote's surrounding section supports, contradicts, or says nothing about an LLM's claim; a string match catches fabricated quotes first."
---

# Cookbook: Double-checking citations

> **TL;DR** Verify LLM citations in two steps: an exact (whitespace/curly-quote normalized) substring match against the source marks quotes that aren't there as `fabricated`; every surviving citation goes to a single `Choice` question — `supports` / `contradicts` / `says_nothing` — over `{"claim": ..., "section": ...}`. Map the choice to a verdict and gate on `confidence >= 0.8` for auto-accept vs. human review.

## Goal

An LLM answers a question and attaches citations: for each claim, a section of a source document and the quote it rests on. Some of those citations are wrong or hallucinated — "the quote can be missing from the document altogether, or sit in it word for word while its context says the opposite of the claim."

The cookbook builds `check_citation()`, which takes a source document plus one citation and returns one of four verdicts: `verified`, `unsupported`, `contradicted`, or `fabricated`, plus a confidence that flags the ones a human should look at.

## Inputs / state shape

The source is [RFC 7519](https://www.rfc-editor.org/rfc/rfc7519.html) (JSON Web Token), committed next to the cookbook as `rfc7519.txt` and split into numbered sections. Citations live in `citations.json`; four are accurate and four were edited to fail the check.

A citation with a quote:

```json
{
  "id": "aud_reject",
  "claim": "If a validator does not find itself in a token's audience list, it has to reject the token.",
  "quote": "If the principal processing the claim does not identify itself with a value in the \"aud\" claim when this claim is present, then the JWT MUST be rejected.",
  "section": "4.1.3"
}
```

A claim-only citation has `"quote": null` and just names a section (e.g. `iat_future`, section `4.1.6`).

The state sent to Jev has exactly two keys:

```python
state={"claim": claim, "section": section}
```

where `section` is the **full text of the section that contains the quote** (found by the string match), not the quote itself. For claim-only citations it is the text of the section the citation names.

Reported corpus stats: `58,365 characters, 45 numbered sections, 8 citations`.

## Questions asked

One `Choice` question, verbatim from the cookbook:

```python
QUESTIONS = {
    "relation": Choice(
        instructions="How does the section relate to the claim?",
        criteria={
            "supports": "The section states the claim or directly implies that it is true",
            "contradicts": "The section states the opposite of the claim or implies it is false",
            "says_nothing": "The section does not address what the claim asserts, either way",
        },
    ),
}
```

`Choice.criteria` is a dict of option name → description in both SDK 0.5.7 and 0.6.0 — only `Score.criteria` changed to an ordered sequence in 0.6.0, so this block is unchanged on 0.6.0.

## Combining logic in code

Setup (cookbook uses `TYPESAFE_MODEL = "jev-1.12"` and `AUTO_ACCEPT = 0.8`, "start high for more human review as you build trust in the model"):

```python
import json, os, re
from pathlib import Path
from time import perf_counter
from typesafe_sdk import Choice, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
AUTO_ACCEPT = 0.8

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)
```

Step 1 — locate the quote (no model involved):

```python
def normalize(text: str) -> str:
    """Collapse whitespace and fold curly quotes, so a quote matches across line wraps."""
    table = str.maketrans({"“": '"', "”": '"', "‘": "'", "’": "'"})
    return re.sub(r"\s+", " ", text.translate(table)).strip()


def find_quote(sections: dict[str, str], quote: str) -> str | None:
    """The number of the section that contains the quote verbatim, or None."""
    needle = normalize(quote)
    for number in sorted(sections, key=lambda n: [int(p) for p in n.split(".")]):
        if needle in normalize(sections[number]):
            return number
    return None


def locate(sections: dict[str, str], citation: dict) -> tuple[str, str | None]:
    """Step 1 for one citation: a status, plus the section step 2 will read."""
    if citation["quote"] is None:
        return "section-only", sections[citation["section"]]
    number = find_quote(sections, citation["quote"])
    if number is None:
        return "missing", None
    return "found", sections[number]
```

Step 2 — ask, then fold both steps into one verdict:

```python
RELATION_TO_VERDICT = {
    "supports": "verified",
    "contradicts": "contradicted",
    "says_nothing": "unsupported",
}


def ask(claim: str, section: str) -> dict:
    started = perf_counter()
    response = client.system_one(
        state={"claim": claim, "section": section},
        questions=QUESTIONS,
        model=TYPESAFE_MODEL,
    )
    answer = response.answers["relation"]
    return {
        "choice": answer.choice,
        "probabilities": answer.probabilities,
        "confidence": answer.confidence,
        "seconds": round(perf_counter() - started, 2),
        "input_tokens": response.usage.input_tokens or 0,
        "output_tokens": response.usage.output_tokens or 0,
    }


def verdict(status: str, answer: dict | None) -> dict:
    """Fold step 1 and step 2 into one of the four labels, plus an auto-or-review flag."""
    if status == "missing":
        # confidence None: no model was called, so there is no model confidence to report
        return {"verdict": "fabricated", "confidence": None, "auto": True}
    return {
        "verdict": RELATION_TO_VERDICT[answer["choice"]],
        "confidence": answer["confidence"],
        "auto": answer["confidence"] >= AUTO_ACCEPT,
    }


def check_citation(sections: dict[str, str], citation: dict) -> dict:
    status, section = locate(sections, citation)
    answer = ask(citation["claim"], section) if section is not None else None
    return {"id": citation["id"], "status": status, "answer": answer, **verdict(status, answer)}
```

The confidence gate: at or above `0.8` the verdict stands on its own; below `0.8` a human confirms it before anything acts on it. See [[patterns/confidence-routing]].

## Results the cookbook reports

Numbers came from `jev-1.12` on 2026-08-16 (TypeSafe's run, replayed from a shipped `json_cache.json`).

Step 1 statuses: seven `found`, one `missing` (`sig_reporting`), one `section-only` (`iat_future`) — the `found` sections ranged from 529 to 3,122 characters.

Full run over the eight citations:

| citation | quote | relation | conf | verdict | action |
|---|---|---|---|---|---|
| epoch_seconds | found | supports | 0.93 | verified | auto |
| aud_reject | found | supports | 0.95 | verified | auto |
| sig_reporting | missing | – | – | fabricated | auto |
| clock_skew | found | supports | 0.99 | verified | auto |
| exp_required | found | contradicts | 0.99 | contradicted | auto |
| pii_encryption | found | says_nothing | 0.27 | unsupported | review |
| iat_future | section-only | says_nothing | 0.56 | unsupported | review |
| duplicate_names | found | supports | 0.99 | verified | auto |

TypeSafe's reading of the run: the four accurate citations all came back `verified` at confidence 0.93 or higher; `sig_reporting` never reached the model because its quote is not in the RFC; `exp_required` quotes section 4.1.4 word for word while the same section says "Use of this claim is OPTIONAL", so it is `contradicted` at 0.99; `pii_encryption` and `iat_future` came back `unsupported` at 0.27 and 0.56, both under the threshold, so both went to a human. `pii_encryption` is the case that shows why the string match is not enough — its quote is in the source verbatim, but its section says nothing about the claim.

## Adapting it to a new domain

- Replace `rfc7519.txt` and `citations.json`. The cookbook states directly: `load_source()` and `split_sections()` are written for an RFC's layout, so a document of another shape needs its own parsing.
- Keep the two-key state (`claim`, `section`). The unit of context is a section, not the whole document — this is what keeps the request small and the judgment atomic.
- The three `criteria` are domain-neutral; they describe a relation between a passage and an assertion, so they transfer to contracts, policies, or papers unchanged (inferred).
- Tune `AUTO_ACCEPT` downward as you measure the model on your own documents; the cookbook's advice is to start high.

## Gotchas

- **`cooksafe` is not publicly installable.** The cookbook's install line is `pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper package on that private index, and `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk` and reimplement the two helpers used here: `JsonCache(Path("json_cache.json"))` is a decorator that memoizes a function's JSON-serializable return value to a file keyed by its arguments (so re-running replays results without calling the API), and `make_playground_link(state, questions, models=[...])` builds a `https://console.typesafe.ai/playground#share/...` URL from a state and a question dict. Neither is needed for the logic; `@json_cache` can be dropped or replaced with `functools.lru_cache`.
- **The string match is exact after normalization.** "A quote that is truncated or lightly reworded comes back as `fabricated`. A production system that tolerates sloppy quoting would need fuzzy matching instead."
- **`confidence` is `None` for fabricated citations** because no model call happened. Anything downstream that reads `confidence` must handle `None`.
- **A verbatim quote is not verification.** `pii_encryption` proves the point: a quote can match the source exactly and still fail to support the claim.
- **Model pinning.** All numbers above are `jev-1.12`; the current default `jev-latest` resolves to `jev-1.13.0`, so re-running live will not reproduce these confidences exactly. Pin the model if you have tuned `AUTO_ACCEPT`.

## Related

- [[cookbooks/overview]] — the cookbook index
- [[concepts/choice]] — the `Choice` primitive and its `criteria`
- [[concepts/confidence]] — what a `0.27` means and what it does not
- [[patterns/confidence-routing]] — the auto-accept / escalate gate
- [[cookbooks/llm-guardrails]] — the other "check an LLM's output" cookbook
- [[cookbooks/classifying-rag-passages]] — judging passages against a query
- [[reference/python-sdk]] — `TypeSafeClient`, `system_one()`, `response.answers`

## Sources

- raw/docs/cookbooks__citation_check.md (https://docs.typesafe.ai/cookbooks/citation_check)