---
title: "Cookbook: Skill suggestion"
type: cookbook
tags: [cookbook, agents, choice, noul, progressive-disclosure]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
- raw/docs/cookbooks__skill_suggestion.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "Two Jev requests per agent turn — rank 182 skills then re-read the top three — cut wrong skill loads from 16.8% to 7.3% and needless loads from 9.8% to 4.0%."
---
# Cookbook: Skill suggestion
> **TL;DR** Request 1 ranks the whole roster with one `Choice` over 182 skill names plus three `Noul` gate questions asking whether the turn needs a skill at all (mean `< 0.30` → suggest nothing). Request 2 re-reads only the top 3, now with full descriptions and 700 characters of each `SKILL.md`, plus one `fits::{name}` `Noul` per candidate that can reject all of them (`max < 0.30` → suggest nothing). The winner's name goes into one extra line of the agent's system prompt.
## Goal
An agent with a large skill roster chooses on almost no information: the roster reaches it as an index with each description truncated (Hermes cuts to 60 characters by default), so the skill that *edits* `.pptx` files reads nearly the same as the one that *authors* them. On a turn where no skill fits, the agent may load one anyway, because a list of names invites a guess.
This cookbook leaves the descriptions alone and uses **progressive disclosure**: read all 182 skills cheaply, then read three of them in detail. The roster itself never changes, so any prefix caching over it still holds.
The output line, verbatim:
```
Relevant to the current request: pptx-author. Ignore this if it does not fit what the user
actually asked for.
```
## Inputs / state shape
**Jev state** is a two-key dict:
```python
def document(request: str) -> dict:
return {"request": request, "recent_context": ""}
```
**Roster.** `hermes_roster.json` holds the 182 skills of [NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) (MIT) at one pinned commit. Each record holds `name`, `category`, `description` (as the index shows it), `description_full`, and `body` (the opening of its `SKILL.md`; the file stores 1600 characters). Measured: **182 skills in 33 categories**, roster prompt **16,089 characters**, index description **54 characters on average, 60 at most**.
**Requests.** `requests.json` holds **488 single-turn requests: 315 covered by exactly one skill (171 distinct skills), 173 covered by none.** The covered ones were written by Claude Sonnet 5 from each skill's own `SKILL.md`, so the labels are trustworthy and the requests are easier than the ones users send. The 173 uncovered ones were all written to punish guessing: 85 everyday requests, 42 technical questions no skill serves (*explain what a monad is*), and 46 that ask for something specific the roster has no skill for, like *post this to Mastodon* on a roster that covers X and nothing else.
**Constants.** `TYPESAFE_MODEL = "jev-1.12"`, `AGENT_MODEL = "claude-haiku-4-5-20251001"`, `SHORTLIST = 3`, `EXCERPT_CHARS = 700`, `GATE_THRESHOLD = 0.30`, `FITS_THRESHOLD = 0.30`, `WORKERS = 8`. Rendered 2026-07-31.
## Questions asked
### Request 1 — rank the whole roster
```python
CHOICE_INSTRUCTIONS = (
"Which of these skills, if any, is the right one to load to help with the "
"user's latest request?"
)
GATE_QUESTIONS = {
"acts_on_user_system": (
"Is the assistant being asked to act on the user's files, accounts, devices, "
"or online services, rather than only to explain or advise?"
),
"would_follow_documented_procedure": (
"Would a careful expert answering this consult a specific documented procedure "
"or set of commands, rather than answering from general understanding?"
),
"prose_suffices": (
"Could a knowledgeable generalist fully satisfy this request in prose, with "
"no tools, no documentation, and no access to the user's files or accounts?"
),
}
INVERTED = {"prose_suffices"} # a yes here points away from needing a skill
```
- `which`: `Choice(instructions=CHOICE_INSTRUCTIONS, criteria={skill["name"]: skill["description"] for skill in ROSTER})` — all 182 names, with the index description as each option's criteria (the same text the agent itself gets). Its probabilities are the ranking.
- `gate::`: one bare `Noul(instructions=text)` per gate question, no criteria.
The cookbook's advice on writing these three: ask whether an **action** is wanted. "A question about subject matter will not separate *explain what a monad is* from a request that needs a skill, since both are software."
### Request 2 — rerank the top three
```python
RERANK_INSTRUCTIONS = (
"Exactly one of these skills is the right one to load for the user's latest "
"request. Which one? Read what each actually does, not just its name."
)
def rerank_criteria(names: tuple[str, ...], excerpt: int) -> dict[str, str]:
return {
name: f"{BY_NAME[name]['description_full']} — {BY_NAME[name]['body'][:excerpt]}"
for name in names
}
def rerank_questions(names: tuple[str, ...], excerpt: int) -> dict:
questions = {
"which": Choice(
instructions=RERANK_INSTRUCTIONS, criteria=rerank_criteria(names, excerpt)
)
}
for name in names:
questions[f"fits::{name}"] = Noul(
instructions=(
f"Does the skill '{name}' do the specific thing the user's request asks "
f"for? It is described as: {BY_NAME[name]['description_full']}"
)
)
return questions
```
Each `fits::{name}` is answered on its own, so they can all come back low — which is what lets the second pass reject an entire shortlist.
## Combining logic in code
```python
import os
from time import perf_counter
from typesafe_sdk import Choice, Noul, TypeSafeClient
TYPESAFE_MODEL = "jev-1.12"
SHORTLIST, EXCERPT_CHARS = 3, 700
GATE_THRESHOLD = FITS_THRESHOLD = 0.30
client = TypeSafeClient(
api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
base_url=os.environ.get("TYPESAFE_ENDPOINT"),
timeout=120.0,
)
def rank_wide(request: str) -> dict:
"""Request 1: rank all 182 skills, and score the request for whether a skill applies."""
questions = {
"which": Choice(
instructions=CHOICE_INSTRUCTIONS,
criteria={skill["name"]: skill["description"] for skill in ROSTER},
)
}
for key, text in GATE_QUESTIONS.items():
questions[f"gate::{key}"] = Noul(instructions=text)
response = client.system_one(
state=document(request), questions=questions, model=TYPESAFE_MODEL
)
ranked = sorted(response.answers["which"].probabilities.items(), key=lambda kv: -kv[1])
values = {
key.removeprefix("gate::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("gate::")
}
oriented = [(1.0 - v) if k in INVERTED else v for k, v in values.items()]
return {"ranked": ranked[:12], "gate": sum(oriented) / len(oriented), "values": values}
def rerank(request: str, names: tuple[str, ...], excerpt: int) -> dict:
"""Request 2: the same Choice over a shortlist, plus one absolute noul per candidate."""
response = client.system_one(
state=document(request),
questions=rerank_questions(names, excerpt),
model=TYPESAFE_MODEL,
)
return {
"winner": response.answers["which"].choice,
"fits": {
key.removeprefix("fits::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("fits::")
},
}
def suggest(request: str) -> tuple[str, ...]:
"""At most one skill name for a request, or () for "nothing here applies"."""
wide = rank_wide(request)
if wide["gate"] < GATE_THRESHOLD:
return ()
shortlist = tuple(name for name, _ in wide["ranked"][:SHORTLIST])
result = rerank(request, shortlist, EXCERPT_CHARS)
if max(result["fits"].values()) < FITS_THRESHOLD:
return ()
return (result["winner"],)
def suggestion_block(names: tuple[str, ...]) -> str:
"""What gets appended after the roster, in the suggestion."""
body = (
f"Relevant to the current request: {', '.join(names)}. Ignore this if it does not "
"fit what the user actually asked for."
if names
else "No skill in the roster appears relevant to this request."
)
return f"\n\n\n{body}\n"
```
The gate is a **mean of three oriented nouls** (`prose_suffices` inverted); the fits check is a **max**. The suggestion goes in its own system-prompt block **after** the roster, past the `cache_control` breakpoint, so the roster text is byte-identical on every turn and prefix caching still holds.
The wording is doing two jobs: it says the suggestion can be ignored, because pushing harder wins compliance on wrong suggestions too and a wrong one is worse than none; and a turn with nothing to suggest still sends a sentence saying so, since sending nothing would leave the roster's own "err on the side of loading" instruction unopposed.
## Results / what the cookbook reports
**Two error metrics, both lower-is-better.** *wrong load*: of covered requests, the share where the first `skill_view` call was not the covering skill (loading nothing counts as a miss). *needless load*: of uncovered requests, the share where the agent called `skill_view` at all.
Over 488 requests against `claude-haiku-4-5-20251001`:
| | loads the wrong skill | loads one when nothing fits |
|---|---|---|
| agent alone, with just its roster | 16.8% | 9.8% |
| **agent with a TypeSafe suggestion** | **7.3%** | **4.0%** |
| agent handed the right answer (oracle) | 2.5% | 1.2% |
`baseline -> TypeSafe: 2.3x fewer wrong loads, 2.4x fewer needless ones`. The oracle row is the ceiling, not a competitor: an agent given the right skill still does not always load it, and no selection method gets past that.
Of the baseline's **36 wrong first picks, 10 came from the right skill's own category** — far more than chance, so the hard part is telling a few lookalikes apart.
**The suggestion helps net, but not monotonically:** `of 315 covered requests: 37 the suggestion fixed, 7 it broke`. "A confident wrong suggestion is more persuasive than no suggestion at all, which is the price of putting one in front of the turn."
**Worked demo, request 1:**
```
"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
needs a skill 0.75 -> suggest (0.31s)
0.990 apple-notes Manage Apple Notes via memo CLI: create, search, edit.
0.010 computer-use Drive the user's desktop in the background — clicking, ty...
"Can you put together a pitch deck skeleton (cover, situation overview, comps, "
needs a skill 0.76 -> suggest (0.16s)
0.700 powerpoint Create, read, edit .pptx decks, slides, notes, templates.
0.300 pptx-author Build PowerPoint decks headless with python-pptx.
"Post this announcement to my Mastodon account."
needs a skill 0.78 -> suggest (0.16s)
0.550 xurl X/Twitter via xurl CLI: raw post search, posting, DM, media.
```
**After request 2:**
```
"Can you save this recipe as a new note ..." was apple-notes -> apple-notes (0.12s)
fits 0.60 apple-notes / 0.54 computer-use / 0.01 concept-diagrams
"Can you put together a pitch deck skeleton ..." was powerpoint -> pptx-author (0.09s)
fits 0.73 powerpoint / 0.38 pptx-author / 0.02 chroma
"Post this announcement to my Mastodon account." was xurl -> xurl (0.09s)
fits 0.56 xurl / 0.38 computer-use / 0.05 openhands
```
The two `.pptx` skills separate once each brings its own text: the deck request flips to the authoring skill. Note the `fits` nouls and the `Choice` disagree there — the nouls score the editing skill higher while the Choice picks the authoring one. **They are deciding different things: the Choice settles *which* skill, the nouls settle *whether* to say anything at all.** The Mastodon request survives both checks and suggests the X skill for a Mastodon request; the second pass can only reject what the wide ranking hands it.
## Adapting it to a new domain
Replace `hermes_roster.json`. Every question reads `name`, `description`, `description_full`, and `body` out of that file, and nothing else knows about Hermes. Then re-tune `GATE_THRESHOLD` and `FITS_THRESHOLD` on labeled turns of your own. The cookbook's summary: "a cheap ranking over everything, then a close look at two or three. Either step may come back empty-handed."
## Gotchas
- **One `Choice` holds 182 options comfortably**, but a few times larger and you would split it into chunks, rank each, then run the shortlist step over the winners.
- **Put the suggestion after the cache breakpoint.** Editing the roster per turn destroys prefix caching; appending a separate system block does not.
- **Always emit a block, even when nothing fits** — silence lets the roster's "err on the side of loading" win.
- **`suggestion_block`'s wording is a measured input, not prose.** It goes to the agent, so it is part of every graded turn's cache key; editing a word silently invalidates the shipped results.
- **A suggestion can break turns the agent had right** (7 of 315 here). Measure both directions.
- **Ranking cannot fix a missing skill.** The Mastodon request has no correct answer on the roster, and both passes still hand back the nearest neighbor.
- **Install line.** `pip install anthropic matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`, plus `TYPESAFE_API_KEY` and `ANTHROPIC_API_KEY`. `cooksafe` is a helper package on TypeSafe's private index, and `pypi.typesafe.ai` returned **404 publicly as of 2026-09-17** — install `pip install typesafe-sdk anthropic` and reimplement the two helpers: `JsonCache(Path("json_cache.json"))`, a decorator that saves each call's result keyed on its inputs so re-running replays the numbers instead of calling either API, and `make_playground_link(state, questions, models=[...])`, which builds a `https://console.typesafe.ai/playground#share/...` URL.
- Published run used `jev-1.12`; this wiki documents `jev-1.13.0`.
## Related
- [[concepts/choice]] — a 182-option ranking
- [[concepts/noul]] — the gate and `fits` questions
- [[concepts/confidence]] — picking the two thresholds
- [[patterns/intent-routing]] — routing to a handler rather than a skill
- [[patterns/fan-out]] — every question in one request
- [[reference/agent-skill]] — TypeSafe's own agent skill and Claude Code plugin
- [[cookbooks/hierarchical-classification]] — the alternative when the option set is a tree
- [[cookbooks/overview]] — the full cookbook catalog
## Sources
- raw/docs/cookbooks__skill_suggestion.md (https://docs.typesafe.ai/cookbooks/skill_suggestion.md)