$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Skill suggestion

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · agents · choice · noul · progressive-disclosure

TL;DR Request 1 ranks the whole roster with one Choice over 182 skill names plus three Noul gate questions asking whether the turn needs a skill at all (mean < 0.30 → suggest nothing). Request 2 re-reads only the top 3, now with full descriptions and 700 characters of each SKILL.md, plus one fits::{name} Noul per candidate that can reject all of them (max < 0.30 → suggest nothing). The winner's name goes into one extra line of the agent's system prompt.

Goal

An agent with a large skill roster chooses on almost no information: the roster reaches it as an index with each description truncated (Hermes cuts to 60 characters by default), so the skill that edits .pptx files reads nearly the same as the one that authors them. On a turn where no skill fits, the agent may load one anyway, because a list of names invites a guess.

This cookbook leaves the descriptions alone and uses progressive disclosure: read all 182 skills cheaply, then read three of them in detail. The roster itself never changes, so any prefix caching over it still holds.

The output line, verbatim:

<skill_relevance>
Relevant to the current request: pptx-author. Ignore this if it does not fit what the user
actually asked for.
</skill_relevance>

Inputs / state shape

Jev state is a two-key dict:

def document(request: str) -> dict:
    return {"request": request, "recent_context": ""}

Roster. hermes_roster.json holds the 182 skills of NousResearch/hermes-agent (MIT) at one pinned commit. Each record holds name, category, description (as the index shows it), description_full, and body (the opening of its SKILL.md; the file stores 1600 characters). Measured: 182 skills in 33 categories, roster prompt 16,089 characters, index description 54 characters on average, 60 at most.

Requests. requests.json holds 488 single-turn requests: 315 covered by exactly one skill (171 distinct skills), 173 covered by none. The covered ones were written by Claude Sonnet 5 from each skill's own SKILL.md, so the labels are trustworthy and the requests are easier than the ones users send. The 173 uncovered ones were all written to punish guessing: 85 everyday requests, 42 technical questions no skill serves (explain what a monad is), and 46 that ask for something specific the roster has no skill for, like post this to Mastodon on a roster that covers X and nothing else.

Constants. TYPESAFE_MODEL = "jev-1.12", AGENT_MODEL = "claude-haiku-4-5-20251001", SHORTLIST = 3, EXCERPT_CHARS = 700, GATE_THRESHOLD = 0.30, FITS_THRESHOLD = 0.30, WORKERS = 8. Rendered 2026-07-31.

Questions asked

Request 1 — rank the whole roster

CHOICE_INSTRUCTIONS = (
    "Which of these skills, if any, is the right one to load to help with the "
    "user's latest request?"
)
GATE_QUESTIONS = {
    "acts_on_user_system": (
        "Is the assistant being asked to act on the user's files, accounts, devices, "
        "or online services, rather than only to explain or advise?"
    ),
    "would_follow_documented_procedure": (
        "Would a careful expert answering this consult a specific documented procedure "
        "or set of commands, rather than answering from general understanding?"
    ),
    "prose_suffices": (
        "Could a knowledgeable generalist fully satisfy this request in prose, with "
        "no tools, no documentation, and no access to the user's files or accounts?"
    ),
}
INVERTED = {"prose_suffices"}  # a yes here points away from needing a skill

The cookbook's advice on writing these three: ask whether an action is wanted. "A question about subject matter will not separate explain what a monad is from a request that needs a skill, since both are software."

Request 2 — rerank the top three

RERANK_INSTRUCTIONS = (
    "Exactly one of these skills is the right one to load for the user's latest "
    "request. Which one? Read what each actually does, not just its name."
)


def rerank_criteria(names: tuple[str, ...], excerpt: int) -> dict[str, str]:
    return {
        name: f"{BY_NAME[name]['description_full']} — {BY_NAME[name]['body'][:excerpt]}"
        for name in names
    }


def rerank_questions(names: tuple[str, ...], excerpt: int) -> dict:
    questions = {
        "which": Choice(
            instructions=RERANK_INSTRUCTIONS, criteria=rerank_criteria(names, excerpt)
        )
    }
    for name in names:
        questions[f"fits::{name}"] = Noul(
            instructions=(
                f"Does the skill '{name}' do the specific thing the user's request asks "
                f"for? It is described as: {BY_NAME[name]['description_full']}"
            )
        )
    return questions

Each fits::{name} is answered on its own, so they can all come back low — which is what lets the second pass reject an entire shortlist.

Combining logic in code

import os
from time import perf_counter

from typesafe_sdk import Choice, Noul, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
SHORTLIST, EXCERPT_CHARS = 3, 700
GATE_THRESHOLD = FITS_THRESHOLD = 0.30

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)


def rank_wide(request: str) -> dict:
    """Request 1: rank all 182 skills, and score the request for whether a skill applies."""
    questions = {
        "which": Choice(
            instructions=CHOICE_INSTRUCTIONS,
            criteria={skill["name"]: skill["description"] for skill in ROSTER},
        )
    }
    for key, text in GATE_QUESTIONS.items():
        questions[f"gate::{key}"] = Noul(instructions=text)
    response = client.system_one(
        state=document(request), questions=questions, model=TYPESAFE_MODEL
    )
    ranked = sorted(response.answers["which"].probabilities.items(), key=lambda kv: -kv[1])
    values = {
        key.removeprefix("gate::"): answer.noul
        for key, answer in response.answers.items()
        if key.startswith("gate::")
    }
    oriented = [(1.0 - v) if k in INVERTED else v for k, v in values.items()]
    return {"ranked": ranked[:12], "gate": sum(oriented) / len(oriented), "values": values}


def rerank(request: str, names: tuple[str, ...], excerpt: int) -> dict:
    """Request 2: the same Choice over a shortlist, plus one absolute noul per candidate."""
    response = client.system_one(
        state=document(request),
        questions=rerank_questions(names, excerpt),
        model=TYPESAFE_MODEL,
    )
    return {
        "winner": response.answers["which"].choice,
        "fits": {
            key.removeprefix("fits::"): answer.noul
            for key, answer in response.answers.items()
            if key.startswith("fits::")
        },
    }


def suggest(request: str) -> tuple[str, ...]:
    """At most one skill name for a request, or () for "nothing here applies"."""
    wide = rank_wide(request)
    if wide["gate"] < GATE_THRESHOLD:
        return ()
    shortlist = tuple(name for name, _ in wide["ranked"][:SHORTLIST])
    result = rerank(request, shortlist, EXCERPT_CHARS)
    if max(result["fits"].values()) < FITS_THRESHOLD:
        return ()
    return (result["winner"],)


def suggestion_block(names: tuple[str, ...]) -> str:
    """What gets appended after the roster, in the suggestion."""
    body = (
        f"Relevant to the current request: {', '.join(names)}. Ignore this if it does not "
        "fit what the user actually asked for."
        if names
        else "No skill in the roster appears relevant to this request."
    )
    return f"\n\n<skill_relevance>\n{body}\n</skill_relevance>"

The gate is a mean of three oriented nouls (prose_suffices inverted); the fits check is a max. The suggestion goes in its own system-prompt block after the roster, past the cache_control breakpoint, so the roster text is byte-identical on every turn and prefix caching still holds.

The wording is doing two jobs: it says the suggestion can be ignored, because pushing harder wins compliance on wrong suggestions too and a wrong one is worse than none; and a turn with nothing to suggest still sends a sentence saying so, since sending nothing would leave the roster's own "err on the side of loading" instruction unopposed.

Results / what the cookbook reports

Two error metrics, both lower-is-better. wrong load: of covered requests, the share where the first skill_view call was not the covering skill (loading nothing counts as a miss). needless load: of uncovered requests, the share where the agent called skill_view at all.

Over 488 requests against claude-haiku-4-5-20251001:

loads the wrong skill loads one when nothing fits
agent alone, with just its roster 16.8% 9.8%
agent with a TypeSafe suggestion 7.3% 4.0%
agent handed the right answer (oracle) 2.5% 1.2%

baseline -> TypeSafe: 2.3x fewer wrong loads, 2.4x fewer needless ones. The oracle row is the ceiling, not a competitor: an agent given the right skill still does not always load it, and no selection method gets past that.

Of the baseline's 36 wrong first picks, 10 came from the right skill's own category — far more than chance, so the hard part is telling a few lookalikes apart.

The suggestion helps net, but not monotonically: of 315 covered requests: 37 the suggestion fixed, 7 it broke. "A confident wrong suggestion is more persuasive than no suggestion at all, which is the price of putting one in front of the turn."

Worked demo, request 1:

"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
  needs a skill 0.75 -> suggest   (0.31s)
    0.990  apple-notes       Manage Apple Notes via memo CLI: create, search, edit.
    0.010  computer-use      Drive the user's desktop in the background — clicking, ty...

"Can you put together a pitch deck skeleton (cover, situation overview, comps, "
  needs a skill 0.76 -> suggest   (0.16s)
    0.700  powerpoint        Create, read, edit .pptx decks, slides, notes, templates.
    0.300  pptx-author       Build PowerPoint decks headless with python-pptx.

"Post this announcement to my Mastodon account."
  needs a skill 0.78 -> suggest   (0.16s)
    0.550  xurl              X/Twitter via xurl CLI: raw post search, posting, DM, media.

After request 2:

"Can you save this recipe as a new note ..."   was apple-notes -> apple-notes   (0.12s)
    fits 0.60  apple-notes / 0.54 computer-use / 0.01 concept-diagrams
"Can you put together a pitch deck skeleton ..." was powerpoint -> pptx-author   (0.09s)
    fits 0.73  powerpoint / 0.38 pptx-author / 0.02 chroma
"Post this announcement to my Mastodon account." was xurl -> xurl   (0.09s)
    fits 0.56  xurl / 0.38 computer-use / 0.05 openhands

The two .pptx skills separate once each brings its own text: the deck request flips to the authoring skill. Note the fits nouls and the Choice disagree there — the nouls score the editing skill higher while the Choice picks the authoring one. They are deciding different things: the Choice settles which skill, the nouls settle whether to say anything at all. The Mastodon request survives both checks and suggests the X skill for a Mastodon request; the second pass can only reject what the wide ranking hands it.

Adapting it to a new domain

Replace hermes_roster.json. Every question reads name, description, description_full, and body out of that file, and nothing else knows about Hermes. Then re-tune GATE_THRESHOLD and FITS_THRESHOLD on labeled turns of your own. The cookbook's summary: "a cheap ranking over everything, then a close look at two or three. Either step may come back empty-handed."

Gotchas

Related

Sources