---
title: "Cookbook: Classification using confidence"
type: cookbook
tags: [cookbook, classification, choice, confidence, taxonomy]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/cookbooks__classification_using_confidence.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "Classify SEC filings into 75 SIC industry groups with one Choice; when confidence < 0.9, report the broader division instead — 40% correct becomes 70%, at no extra call."
---

# Cookbook: Classification using confidence

> **TL;DR** One `Choice` over 75 SIC major groups per 10-K "Item 1. Business" section. Read `answer.confidence` (not the winner's probability). At `>= 0.9`, report the group; below, report the *division* that group rolls up into — a free coarser label from the same response. Across 60 filings: 30 confident answers were 90% right, 30 unconfident ones 40% right as groups but 70% right as divisions.

## Goal

Classify company self-descriptions under the Standard Industrial Classification. "Most filings are easy. A regional bank is a regional bank. Some are not: a company that just sold one of its two segments, or a startup describing a business it plans to enter rather than one it runs. The model has to pick a group regardless, and the answer for a hard case looks no different from the answer for an easy one. Telling hard cases from easy ones is normally where the cost goes: a second model, extra calls, human review."

The recipe's insight: a Choice already tells you, via `confidence`. And because SIC labels form a hierarchy, the fallback costs nothing — "The broad label follows from the narrow one, so there is no second call."

The deliverable is `classify()`, returning a label plus how specific it is, at one request per document.

## Inputs / state shape

The state is the **raw text of Item 1 "Business"** — a bare string, not a dict:

```python
response = client.system_one(state=text, questions=questions(), model=TYPESAFE_MODEL)
```

Two data files:

- `sic_codes.tsv` — the SEC's own industry list, fetched 2026-08-10: 444 four-digit codes with titles. Reported rollup: `444 industries -> 75 major groups -> 10 divisions`.
- `filings.jsonl` — 60 annual reports (10-K) trimmed to Item 1, spanning 1993–2024, 700 to 2,200 words each (`60 filings, 1438 words on average`). Each carries the SIC code its filer chose plus an EDGAR accession number.

On the gold labels, stated plainly by the cookbook: "It is self-reported: whoever prepared the filing picked it once, and it goes stale when a company sells the business the code names and keeps the code. These 60 were filtered down to filings whose own text supports the code they carry, so the numbers here measure the recipe rather than the state of EDGAR's metadata."

The taxonomy is built with no model involved — group the four-digit codes by their first two digits, then map those digits to a division:

```python
DIVISIONS = [
    (1, 9, "agriculture, forestry and fishing"),
    (10, 14, "mining"),
    (15, 17, "construction"),
    (20, 39, "manufacturing"),
    (40, 49, "transportation, communications and utilities"),
    (50, 51, "wholesale trade"),
    (52, 59, "retail trade"),
    (60, 67, "finance, insurance and real estate"),
    (70, 89, "services"),
    (91, 99, "public administration"),
]


def division(group: str) -> str:
    number = int(group)
    return next(name for low, high, name in DIVISIONS if low <= number <= high)
```

### Describing 75 options

"A Choice question needs something to describe each option, and a group's own name is not always there: 42 of the 75 carry an umbrella title in the SEC's list, and the rest carry none. So each group is described by the industries inside it, which is what someone reading the filing would match against anyway."

```python
MAX_NAMED = 8  # industries listed per group; enough to characterise it without a wall of text


def describe(group: str) -> str:
    umbrella = INDUSTRIES.get(f"{group}00")
    inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
    listed = "; ".join(inside)
    return f"{umbrella} — includes: {listed}" if umbrella and listed else (umbrella or listed)
```

Sample output: `group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product...`

## Questions asked

One `Choice`, verbatim:

```python
QUESTION = (
    "Which broad industry does this company operate in? Judge the company's own operations "
    "as this filing describes them."
)


def questions() -> dict:
    return {
        "group": Choice(
            instructions=QUESTION,
            criteria={group: describe(group) for group in sorted(GROUPS)},
        )
    }
```

The cookbook's capacity note: "The whole taxonomy fits in one request: a Choice works reliably up to roughly 240 options, and 75 is well inside that."

`Choice.criteria` remains a mapping in SDK 0.6.0; only `Score.criteria` became an ordered sequence.

## Combining logic in code

```python
import os
from typesafe_sdk import Choice, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
CONFIDENT = 0.9  # above this the group is reported; below it, the division

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)


def ask(filing_id: str, text: str) -> dict:
    response = client.system_one(state=text, questions=questions(), model=TYPESAFE_MODEL)
    answer = response.answers["group"]
    return {
        "group": answer.choice,
        "confidence": answer.confidence,
        "probabilities": dict(answer.probabilities),
    }


def classify(filing: dict) -> dict:
    answer = ask(filing["id"], filing["text"])
    sure = answer["confidence"] >= CONFIDENT
    return {
        "level": "group" if sure else "division",
        "label": answer["group"] if sure else division(answer["group"]),
        "confidence": answer["confidence"],
        "group": answer["group"],
    }
```

Scoring both policies against the filer's own code:

```python
def correct(filing: dict, result: dict) -> bool:
    gold_group = filing["sic"][:2]
    if result["level"] == "group":
        return result["label"] == gold_group
    return result["label"] == division(gold_group)
```

Why `confidence` rather than the winner's probability: "A winner at 0.45 with a runner-up at 0.44, and a winner at 0.45 with the rest of the weight scattered thinly, are different situations, and `confidence` is what separates them." See [[concepts/confidence]].

Note that every filing still gets a usable label: "One the model could not classify confidently comes back one level up instead of being dropped or sent on. If a division is too coarse for your application to act on, this branch is where you hand it to a person."

## Results the cookbook reports

Numbers came from `jev-1.12` on 2026-08-12.

Extremes of the confidence distribution:

```
three filings the model was sure about:
    310158_1996  conf 1.00  -> group    28             (group 28: chemicals & allied products)
     33416_1998  conf 1.00  -> group    63             (group 63: life insurance; accident & health insurance; h)
    352541_1996  conf 1.00  -> group    49             (group 49: electric, gas & sanitary services)

three it was not:
   1372167_2013  conf 0.22  -> division manufacturing  (group 38: search, detection, navagation, guidance, aeron)
   1398633_2009  conf 0.23  -> division wholesale trade (group 50: wholesale-durable goods)
     46653_1999  conf 0.29  -> division services       (group 87: services-engineering, accounting, research, ma)
```

"The three at 1.00 are a pharmaceutical maker, a life insurer and a utility; all three are holding companies on paper, but each has one dominant business the filing names outright." The three at the bottom: two are development-stage companies describing a business they intend to start (one "intends to operate as a software developer", another was "organized to enter into the computer security software industry"), and the third had two segments and sold one of them weeks before filing.

The headline table over all 60:

```
forced to name a group every time      39/60 right
  of those, the 30 it was sure about  27/30 right
  and the 30 it was not           12/30 right

letting it answer coarsely when unsure  48/60 useful answers
```

| population | n | group named | division named |
|---|---|---|---|
| confidence >= 0.9 | 30 | 27/30 (90%) | — (group reported) |
| confidence < 0.9 | 30 | 12/30 (40%) | 70% |
| all | 60 | 39/60 | 48/60 useful answers |

A confidence cutoff of 0.9 split the 60 filings exactly in half on this set.

## Adapting it to a new domain

- "Point `ask()` at your own documents and rewrite `describe()` for your own taxonomy, and the rest carries over."
- The pattern needs a hierarchy — or any coarser grouping of the same options — that is derivable from the fine label in code. `DIVISIONS` is a hand-written range table; nothing about it is model-facing.
- Where an option has no usable name of its own, describe it by its members, as `describe()` does. That is what a human reader would match against.
- Tune `CONFIDENT` against your own accuracy-vs-specificity trade-off; `0.9` here is a choice, not a default.
- With no hierarchy, the same `confidence` branch can route to a human or a slower model instead — see [[patterns/confidence-routing]] and [[cookbooks/sde-cascade]].
- For deeper taxonomies where you want to walk down level by level, see [[cookbooks/hierarchical-classification]].

## Gotchas

- **`cooksafe` is not publicly installable.** Install line: `pip install ipython matplotlib "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper on a private index; `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk` and reimplement what you need — `JsonCache(Path("json_cache.json"))` is a decorator memoizing JSON-serializable returns to a file keyed by the arguments; `make_playground_link(state, questions, models=[...])` builds a `console.typesafe.ai/playground#share/...` URL.
- **Option-count ceiling.** The cookbook states a Choice "works reliably up to roughly 240 options." The launch blog separately states Jev "supports a cardinality up to 255." Both numbers come from TypeSafe; treat ~240 as the practical working limit and split the taxonomy above it.
- **Confidence != the top probability.** Reading `probabilities[choice]` instead of `confidence` would collapse the two cases the recipe depends on telling apart.
- **The 90% / 40% figures are for one 60-filing set, hand-filtered so the self-reported label is supported by the text.** They are not an accuracy claim about SIC classification in the wild.
- **A division answer may still be wrong** — 70%, not 100%. If a coarse label can't be acted on, treat that branch as escalation, not as an answer.
- **Model pinning.** All numbers are `jev-1.12`; `jev-latest` now resolves to `jev-1.13.0`, so a live re-run will shift both the confidences and the 0.9 split.

## Related

- [[cookbooks/overview]] — the cookbook index
- [[concepts/confidence]] — calibration, and what a 0.9 cutoff means
- [[concepts/choice]] — options, `criteria`, `probabilities`
- [[patterns/confidence-routing]] — the general gate
- [[cookbooks/hierarchical-classification]] — walking a taxonomy level by level
- [[cookbooks/sde-cascade]] — escalating the unconfident tail to a bigger model
- [[reference/python-sdk-responses]] — `answers`, `confidence`, `probabilities`, `usage`

## Sources

- raw/docs/cookbooks__classification_using_confidence.md (https://docs.typesafe.ai/cookbooks/classification_using_confidence)