$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Autoresearch feature discovery

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · autoresearch · feature-engineering · score · noul · catboost

TL;DR Turn free text into a numeric feature table without hand-writing features. An LLM proposes add / revise / drop actions against a fixed schema; each proposal becomes a Score (five fixed intensity levels → 2 columns: expected level + spread) or a Noul (1 column: P(true)); Jev answers all of a round's questions in one request per row; CatBoost cross-validates, and its importances plus its worst-predicted rows feed the next proposal. Five rounds, 2,000 wine reviews: 38 questions → 67 columns → held-out RMSE 1.77 vs 3.09 (mean baseline), 2.47 (word counts), 2.15 (asking Jev for the score directly).

Goal

"CatBoost needs a table of numbers, and a tasting note is not one. This cookbook builds the table out of questions about the note, and none of them are written by hand."

The autoresearch part: "CatBoost reports which questions it used and which rows it still gets wrong, the following proposal call reads that report, and the loop runs again."

Inputs / state shape

Dataset: the pinned GroNLP/ik-nlp-22_winemag CSV — 2,000 deduplicated wine reviews, description → points (80–100 critic scale). Split N_DEV, N_TEST = 1200, 800; the loop reads dev labels only, test is scored once at the end.

1200 dev rows, 800 held out; scores run 80-98, mean 88.73, sd 3.17

The state Jev receives is the bare note string — client.system_one(state=note, questions=..., model=...). Nothing else.

Configuration, verbatim:

TYPESAFE_MODEL = "jev-1.12"
N_DEV, N_TEST = 1200, 800   # the loop reads dev labels only; test is scored once
ROUNDS = 5                  # a round answers questions for all 2,000 rows: 2,000 requests
PROPOSER = "claude-sonnet-5"  # or "gpt-5.6-luna"; the cache holds the Anthropic run
EXAMPLES = 60               # dev notes the proposer reads per round, half of them its worst misses
PROPOSALS = 18              # actions the proposer may return per round
MIN_SPREAD = 0.05           # a column this flat cannot separate anything, so it is not kept
CHANGE_TOLERANCE = 0.0      # a revision or drop has to improve dev error, not just not hurt
ENCODING = "mean_spread"    # a score answer becomes two columns: its mean and spread
FOLDS, REPEATS = 5, 3       # repeats steady the error at this sample size
CATBOOST = dict(iterations=400, depth=4, learning_rate=0.05, loss_function="RMSE",
                verbose=0, random_seed=0, thread_count=1, allow_writing_files=False)

Questions asked

Two fixed rubrics turn any proposed question into a Jev question. Verbatim:

INTENSITY_LEVELS = [
    "Not present in this note at all",
    "Barely present - mentioned once, in passing",
    "Present at a moderate level",
    "Present strongly - the note dwells on it",
    "Dominant - the note is largely about this",
]
PRESENCE_CRITERIA = NoulCriteria(
    true="The note states this or clearly implies it",
    false="The note gives no indication of this",
)


def feature_questions(features: list[dict]) -> dict:
    questions = {}
    for feature in features:
        if feature["kind"] == "intensity":
            questions[feature["name"]] = Score(
                instructions=feature["question"], criteria=INTENSITY_LEVELS
            )
        else:
            questions[feature["name"]] = Noul(
                instructions=feature["question"], criteria=PRESENCE_CRITERIA
            )
    return questions

Score(criteria=INTENSITY_LEVELS) is already the SDK 0.6.0 ordered-sequence form; no migration needed.

The instructions are written by the proposer LLM at runtime, so they are not fixed. The one the loop ranked highest, quoted from the run:

note_overall_tone_positivity: "Setting aside specific descriptors, how positive is the overall emotional tone and word choice of the note taken as a whole (warm, admiring language throughout vs. flat, neutral, or lukewarm phrasing)?"

The shortcut baseline is one Score over ten bands:

SCORE_LEVELS = [
    "Faulty or unpleasant - the note is mostly criticism",
    "Barely acceptable - drinkable, with nothing to recommend it",
    "Simple and sound - correct, plain, forgettable",
    "Pleasant everyday wine - some appeal, little depth",
    "Good - clear varietal character, well made",
    "Very good - balanced, with something to say",
    "Excellent - complex and structured",
    "Outstanding - depth and length, built to age",
    "Superb - among the best of its type",
    "Profound - the note treats it as exceptional",
]

# instructions:
"Judging only by what this tasting note says, how good is the wine?"

The proposer brief

The only domain-specific string in the file. Abridged head, verbatim:

You are designing numeric features for a gradient-boosting model that
predicts the score a wine critic gave (an integer from 80 to 100) from the tasting note alone.
The model sees nothing but the features you design.

Return up to 18 actions. Each action is one of:

- {"op": "add", "target": "", "name": ..., "kind": ..., "question": ...}
- {"op": "revise", "target": <name of an existing feature>, "name": ..., "kind": ..., "question": ...}
- {"op": "drop", "target": <name of an existing feature>, "name": "", "kind": "intensity", "question": ""}

`kind` is "intensity" for something with a degree, or "presence" for a yes/no fact.
`question` is what gets asked about one tasting note.
...
Good features can be judged from the note's own words, vary from note to note, and carry
information about quality that the other features do not.

Constrained by PROPOSAL_SCHEMA, a JSON Schema with op ∈ {add, revise, drop}, additionalProperties: False, and every field in required — with the note: "Structured output requires every property in required, so unused fields come back empty."

Combining logic in code

Answer one round's questions

def answer(model: str, note: str, features_json: str) -> dict:
    """One request per note; every question of the round rides it. Keeps every probability."""
    features = json.loads(features_json)
    response = client.system_one(
        state=note, questions=feature_questions(features), model=model
    )
    raw = {}
    for feature in features:
        got = response.answers[feature["name"]]
        if feature["kind"] == "intensity":
            raw[feature["name"]] = [
                got.probabilities.get(i, 0.0) for i in range(len(INTENSITY_LEVELS))
            ]
        else:
            raw[feature["name"]] = [got.noul]
    return {"raw": raw,
            "input_tokens": response.usage.input_tokens or 0,
            "output_tokens": response.usage.output_tokens or 0}


def featurize(notes: list[str], features: list[dict]) -> dict:
    """Answer one question set for many notes: one request each, eight in flight."""
    payload = json.dumps(features, sort_keys=True)
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(lambda note: answer(TYPESAFE_MODEL, note, payload), notes))
    return {f["name"]: np.array([r["raw"][f["name"]] for r in results], dtype=float)
            for f in features}

Note it keeps the full probability vector of a Score, not just .score.

Encode probabilities as columns

def encode(feature: dict, probabilities: np.ndarray, mode: str) -> list[tuple]:
    """Turn one question's probabilities into named columns."""
    name = feature["name"]
    if feature["kind"] == "presence":
        return [(name, probabilities[:, 0])]  # one number is all there is
    levels = np.arange(probabilities.shape[1])
    mean = probabilities @ levels
    if mode == "mean":
        return [(name, mean)]
    if mode == "mean_spread":
        variance = probabilities @ (levels**2) - mean**2
        return [(name, mean), (f"{name}_sd", np.sqrt(np.clip(variance, 0, None)))]
    return [(f"{name}_p{i}", probabilities[:, i]) for i in levels]

Three encodings are available; the run uses mean_spread. Column budget: 29 score × 2 + 9 noul × 1 = 67.

Accept / reject rules

def try_change(trial, accepted, cv, answers_for, split, mode, tolerance):
    """Refit with the change and keep it only if the dev error improves. No API calls."""
    _, cv_trial = evaluate(trial, answers_for, split, mode)
    if cv_trial <= cv + tolerance:
        return trial, cv_trial, f"CV {cv:.3f} -> {cv_trial:.3f}", True
    return accepted, cv, f"would cost {cv_trial - cv:+.3f}", False

Asymmetric on purpose: "An added question goes straight in: its answers have already been fetched, and its importance will show later whether it was worth asking. A revision or a drop takes away a column the model is already using, so each one is tried first… A refit costs no API calls, so trying a change and rejecting it is free." The only filter on an add is column[split.dev].std() < MIN_SPREAD → journaled as flat.

Which rows the next round reads

def example_rows(split, out_of_fold, n):
    """Select representative dev rows for a proposer round."""
    dev = split.dev
    if out_of_fold is None:
        ranked = dev[np.argsort(split.scores[dev], kind="stable")]
        return [int(ranked[round(q * (len(ranked) - 1))]) for q in np.linspace(0, 1, n)]
    error = np.abs(split.scores[dev] - out_of_fold)
    order = np.argsort(-error, kind="stable")
    worst = [int(dev[i]) for i in order[: n // 2]]
    best = [int(dev[i]) for i in order[len(order) - (n - n // 2) :]]
    return worst + best

Round 1 spans the score range; later rounds send 30 worst + 30 best, framed for the proposer as: "The first half is where your current questions miss by the most and the second half is where they are already right, so what separates the halves is what the questions have not captured."

The feedback scoreboard

feedback_for() builds the text the next proposal reads: CV RMSE per round so far; how many dev notes moved by more than 0.1 points vs. the previous round; and per feature, "importance as a percentage of the total and the spread of the column across the dev rows. Low importance or low spread means the question is not doing much; revise or drop it." All numbers are rounded before entering the prompt, so a replay hits the cache.

The loop, in pseudocode (verbatim from the cookbook)

questions <- {}
repeat for each round:
    notes  <- round 1 ? 60 dev notes across the score range
                      : the 30 worst-predicted dev notes + the 30 best,
                        each with its score, this prediction and the last
    actions <- LLM(brief, questions, notes, importance and error so far)
    answers[q] <- TypeSafe(note, all new questions of this round) for every row
    for each added q:      keep it unless its column is flat
    for each revised q:    refit; keep the change only if dev error drops
    for each dropped q:    refit; drop it only if dev error drops
    out_of_fold <- k-fold CatBoost on the columns   # judges, and picks next round's notes

A key cost property: "No question is filtered out before it is answered. All of a round's questions go out in the same request, so one more question costs no extra request. A question that applies to one row in ten will look useless in the 60 notes the proposer reads, and still be the most useful column in the set."

Results the cookbook reports

Numbers from TypeSafe jev-1.12 and claude-sonnet-5 on 2026-08-03, scored once on the 800 held-out rows.

arm RMSE Spearman
predict the mean of the dev rows 3.088 -0.014
the note as word counts, same CatBoost 2.466 0.605
ask for the score itself, shifted -1.71 2.145 0.761
18 questions from round 1, no loop 1.869 0.778
38 questions after all 5 rounds 1.772 0.799

Per-round dev CV RMSE and set size: round 1 → 18 features, 1.903; round 2 → 23, 1.881; round 3 → 30, 1.861; round 4 → 35, 1.838; round 5 → 38, 1.840.

The gain attributable to rounds 2–5, bootstrapped over 2,000 resamples of the held-out rows:

round 1 -> round 5 on the held-out rows: -0.097 points, 95% CI [-0.147, -0.050]

"Most of the gain is in that first call." Also: "Round 5 proposed four adds, two rewordings and eight drops, and gave the first dev number that did not improve. There is only so much to ask about a 245-character note, and by round 5 the proposals had tipped from adding questions to dropping them."

Final importance shares (CatBoost importance normalized so all 38 questions sum to 100%; a score question's two columns summed back together):

feature                               asked as  importance share
note_overall_tone_positivity          score         17.4%
savory_food_wine_seriousness          score          8.7%
positive_superlative_language         score          8.4%
single_vineyard_or_prestige_signal    noul           7.2%
descriptive_detail_density            score          5.7%
elegance_finesse_language             score          5.0%
complexity                            score          5.0%
aging_potential                       score          5.0%
balance_harmony                       score          2.9%
drinkability_easiness                 score          2.9%
critic_enthusiasm_confidence          score          2.7%
flavor_distinctiveness                score          2.6%

the 38 questions the loop kept: 29 score, 9 noul

TypeSafe's own scoping of the claim: "The word-count row is CatBoost's own text handling, not a tuned text-regression pipeline. All of this is one dataset and one run of the loop." And on the dev-vs-held-out gap: "The dev line runs above the held-out line the whole way, and that is a training-size effect."

Adapting it to a new domain

Gotchas

Related

Sources