--- title: "Cookbook: Autoresearch feature discovery" type: cookbook tags: [cookbook, autoresearch, feature-engineering, score, noul, catboost] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__autoresearch_feature_discovery.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "An LLM proposes Jev questions, Jev answers them per row to build numeric columns, CatBoost trains and reports back: 38 questions after five rounds reach 1.77 held-out RMSE on wine scores." --- # Cookbook: Autoresearch feature discovery > **TL;DR** Turn free text into a numeric feature table without hand-writing features. An LLM proposes `add` / `revise` / `drop` actions against a fixed schema; each proposal becomes a `Score` (five fixed intensity levels → 2 columns: expected level + spread) or a `Noul` (1 column: P(true)); Jev answers **all of a round's questions in one request per row**; CatBoost cross-validates, and its importances plus its worst-predicted rows feed the next proposal. Five rounds, 2,000 wine reviews: 38 questions → 67 columns → **held-out RMSE 1.77** vs 3.09 (mean baseline), 2.47 (word counts), 2.15 (asking Jev for the score directly). ## Goal "CatBoost needs a table of numbers, and a tasting note is not one. This cookbook builds the table out of questions about the note, and none of them are written by hand." The autoresearch part: "CatBoost reports which questions it used and which rows it still gets wrong, the following proposal call reads that report, and the loop runs again." ## Inputs / state shape Dataset: the pinned `GroNLP/ik-nlp-22_winemag` CSV — 2,000 deduplicated wine reviews, `description` → `points` (80–100 critic scale). Split `N_DEV, N_TEST = 1200, 800`; the loop reads dev labels only, test is scored once at the end. ``` 1200 dev rows, 800 held out; scores run 80-98, mean 88.73, sd 3.17 ``` **The state Jev receives is the bare note string** — `client.system_one(state=note, questions=..., model=...)`. Nothing else. Configuration, verbatim: ```python TYPESAFE_MODEL = "jev-1.12" N_DEV, N_TEST = 1200, 800 # the loop reads dev labels only; test is scored once ROUNDS = 5 # a round answers questions for all 2,000 rows: 2,000 requests PROPOSER = "claude-sonnet-5" # or "gpt-5.6-luna"; the cache holds the Anthropic run EXAMPLES = 60 # dev notes the proposer reads per round, half of them its worst misses PROPOSALS = 18 # actions the proposer may return per round MIN_SPREAD = 0.05 # a column this flat cannot separate anything, so it is not kept CHANGE_TOLERANCE = 0.0 # a revision or drop has to improve dev error, not just not hurt ENCODING = "mean_spread" # a score answer becomes two columns: its mean and spread FOLDS, REPEATS = 5, 3 # repeats steady the error at this sample size CATBOOST = dict(iterations=400, depth=4, learning_rate=0.05, loss_function="RMSE", verbose=0, random_seed=0, thread_count=1, allow_writing_files=False) ``` ## Questions asked Two fixed rubrics turn any proposed question into a Jev question. Verbatim: ```python INTENSITY_LEVELS = [ "Not present in this note at all", "Barely present - mentioned once, in passing", "Present at a moderate level", "Present strongly - the note dwells on it", "Dominant - the note is largely about this", ] PRESENCE_CRITERIA = NoulCriteria( true="The note states this or clearly implies it", false="The note gives no indication of this", ) def feature_questions(features: list[dict]) -> dict: questions = {} for feature in features: if feature["kind"] == "intensity": questions[feature["name"]] = Score( instructions=feature["question"], criteria=INTENSITY_LEVELS ) else: questions[feature["name"]] = Noul( instructions=feature["question"], criteria=PRESENCE_CRITERIA ) return questions ``` `Score(criteria=INTENSITY_LEVELS)` is already the SDK 0.6.0 ordered-sequence form; no migration needed. The instructions are **written by the proposer LLM at runtime**, so they are not fixed. The one the loop ranked highest, quoted from the run: > `note_overall_tone_positivity`: "Setting aside specific descriptors, how positive is the overall emotional tone and word choice of the note taken as a whole (warm, admiring language throughout vs. flat, neutral, or lukewarm phrasing)?" The shortcut baseline is one `Score` over ten bands: ```python SCORE_LEVELS = [ "Faulty or unpleasant - the note is mostly criticism", "Barely acceptable - drinkable, with nothing to recommend it", "Simple and sound - correct, plain, forgettable", "Pleasant everyday wine - some appeal, little depth", "Good - clear varietal character, well made", "Very good - balanced, with something to say", "Excellent - complex and structured", "Outstanding - depth and length, built to age", "Superb - among the best of its type", "Profound - the note treats it as exceptional", ] # instructions: "Judging only by what this tasting note says, how good is the wine?" ``` ### The proposer brief The only domain-specific string in the file. Abridged head, verbatim: ``` You are designing numeric features for a gradient-boosting model that predicts the score a wine critic gave (an integer from 80 to 100) from the tasting note alone. The model sees nothing but the features you design. Return up to 18 actions. Each action is one of: - {"op": "add", "target": "", "name": ..., "kind": ..., "question": ...} - {"op": "revise", "target": , "name": ..., "kind": ..., "question": ...} - {"op": "drop", "target": , "name": "", "kind": "intensity", "question": ""} `kind` is "intensity" for something with a degree, or "presence" for a yes/no fact. `question` is what gets asked about one tasting note. ... Good features can be judged from the note's own words, vary from note to note, and carry information about quality that the other features do not. ``` Constrained by `PROPOSAL_SCHEMA`, a JSON Schema with `op` ∈ `{add, revise, drop}`, `additionalProperties: False`, and every field in `required` — with the note: "Structured output requires every property in `required`, so unused fields come back empty." ## Combining logic in code ### Answer one round's questions ```python def answer(model: str, note: str, features_json: str) -> dict: """One request per note; every question of the round rides it. Keeps every probability.""" features = json.loads(features_json) response = client.system_one( state=note, questions=feature_questions(features), model=model ) raw = {} for feature in features: got = response.answers[feature["name"]] if feature["kind"] == "intensity": raw[feature["name"]] = [ got.probabilities.get(i, 0.0) for i in range(len(INTENSITY_LEVELS)) ] else: raw[feature["name"]] = [got.noul] return {"raw": raw, "input_tokens": response.usage.input_tokens or 0, "output_tokens": response.usage.output_tokens or 0} def featurize(notes: list[str], features: list[dict]) -> dict: """Answer one question set for many notes: one request each, eight in flight.""" payload = json.dumps(features, sort_keys=True) with ThreadPoolExecutor(max_workers=8) as pool: results = list(pool.map(lambda note: answer(TYPESAFE_MODEL, note, payload), notes)) return {f["name"]: np.array([r["raw"][f["name"]] for r in results], dtype=float) for f in features} ``` Note it keeps the **full probability vector** of a Score, not just `.score`. ### Encode probabilities as columns ```python def encode(feature: dict, probabilities: np.ndarray, mode: str) -> list[tuple]: """Turn one question's probabilities into named columns.""" name = feature["name"] if feature["kind"] == "presence": return [(name, probabilities[:, 0])] # one number is all there is levels = np.arange(probabilities.shape[1]) mean = probabilities @ levels if mode == "mean": return [(name, mean)] if mode == "mean_spread": variance = probabilities @ (levels**2) - mean**2 return [(name, mean), (f"{name}_sd", np.sqrt(np.clip(variance, 0, None)))] return [(f"{name}_p{i}", probabilities[:, i]) for i in levels] ``` Three encodings are available; the run uses `mean_spread`. Column budget: `29 score × 2 + 9 noul × 1 = 67`. ### Accept / reject rules ```python def try_change(trial, accepted, cv, answers_for, split, mode, tolerance): """Refit with the change and keep it only if the dev error improves. No API calls.""" _, cv_trial = evaluate(trial, answers_for, split, mode) if cv_trial <= cv + tolerance: return trial, cv_trial, f"CV {cv:.3f} -> {cv_trial:.3f}", True return accepted, cv, f"would cost {cv_trial - cv:+.3f}", False ``` Asymmetric on purpose: "An added question goes straight in: its answers have already been fetched, and its importance will show later whether it was worth asking. A revision or a drop takes away a column the model is already using, so each one is tried first… A refit costs no API calls, so trying a change and rejecting it is free." The only filter on an add is `column[split.dev].std() < MIN_SPREAD` → journaled as `flat`. ### Which rows the next round reads ```python def example_rows(split, out_of_fold, n): """Select representative dev rows for a proposer round.""" dev = split.dev if out_of_fold is None: ranked = dev[np.argsort(split.scores[dev], kind="stable")] return [int(ranked[round(q * (len(ranked) - 1))]) for q in np.linspace(0, 1, n)] error = np.abs(split.scores[dev] - out_of_fold) order = np.argsort(-error, kind="stable") worst = [int(dev[i]) for i in order[: n // 2]] best = [int(dev[i]) for i in order[len(order) - (n - n // 2) :]] return worst + best ``` Round 1 spans the score range; later rounds send 30 worst + 30 best, framed for the proposer as: "The first half is where your current questions miss by the most and the second half is where they are already right, so what separates the halves is what the questions have not captured." ### The feedback scoreboard `feedback_for()` builds the text the next proposal reads: CV RMSE per round so far; how many dev notes moved by more than 0.1 points vs. the previous round; and per feature, "importance as a percentage of the total and the spread of the column across the dev rows. Low importance or low spread means the question is not doing much; revise or drop it." All numbers are rounded before entering the prompt, so a replay hits the cache. ### The loop, in pseudocode (verbatim from the cookbook) ``` questions <- {} repeat for each round: notes <- round 1 ? 60 dev notes across the score range : the 30 worst-predicted dev notes + the 30 best, each with its score, this prediction and the last actions <- LLM(brief, questions, notes, importance and error so far) answers[q] <- TypeSafe(note, all new questions of this round) for every row for each added q: keep it unless its column is flat for each revised q: refit; keep the change only if dev error drops for each dropped q: refit; drop it only if dev error drops out_of_fold <- k-fold CatBoost on the columns # judges, and picks next round's notes ``` A key cost property: "No question is filtered out before it is answered. All of a round's questions go out in the same request, so one more question costs no extra request. A question that applies to one row in ten will look useless in the 60 notes the proposer reads, and still be the most useful column in the set." ## Results the cookbook reports Numbers from TypeSafe `jev-1.12` and `claude-sonnet-5` on 2026-08-03, scored once on the 800 held-out rows. | arm | RMSE | Spearman | |---|---|---| | predict the mean of the dev rows | 3.088 | -0.014 | | the note as word counts, same CatBoost | 2.466 | 0.605 | | ask for the score itself, shifted -1.71 | 2.145 | 0.761 | | 18 questions from round 1, no loop | 1.869 | 0.778 | | 38 questions after all 5 rounds | **1.772** | **0.799** | Per-round dev CV RMSE and set size: round 1 → 18 features, 1.903; round 2 → 23, 1.881; round 3 → 30, 1.861; round 4 → 35, 1.838; round 5 → 38, 1.840. The gain attributable to rounds 2–5, bootstrapped over 2,000 resamples of the held-out rows: ``` round 1 -> round 5 on the held-out rows: -0.097 points, 95% CI [-0.147, -0.050] ``` "Most of the gain is in that first call." Also: "Round 5 proposed four adds, two rewordings and eight drops, and gave the first dev number that did not improve. There is only so much to ask about a 245-character note, and by round 5 the proposals had tipped from adding questions to dropping them." Final importance shares (CatBoost importance normalized so all 38 questions sum to 100%; a score question's two columns summed back together): ``` feature asked as importance share note_overall_tone_positivity score 17.4% savory_food_wine_seriousness score 8.7% positive_superlative_language score 8.4% single_vineyard_or_prestige_signal noul 7.2% descriptive_detail_density score 5.7% elegance_finesse_language score 5.0% complexity score 5.0% aging_potential score 5.0% balance_harmony score 2.9% drinkability_easiness score 2.9% critic_enthusiasm_confidence score 2.7% flavor_distinctiveness score 2.6% the 38 questions the loop kept: 29 score, 9 noul ``` TypeSafe's own scoping of the claim: "The word-count row is CatBoost's own text handling, not a tuned text-regression pipeline. All of this is one dataset and one run of the loop." And on the dev-vs-held-out gap: "The dev line runs above the held-out line the whole way, and that is a training-size effect." ## Adapting it to a new domain - "`PROPOSER_TASK` is the only string that mentions wine, and `featurize()` takes any list of strings." Rewrite the brief with your label, its range, and what your reviewers/authors actually vary on. - Editing the brief changes the cache key, "so the next run calls the API again for every round." Budget for that. - Keep the two fixed rubrics (`INTENSITY_LEVELS`, `PRESENCE_CRITERIA`) unless your text needs different graduations — they are what lets an arbitrary proposed question become a comparable number. - `ENCODING` is a lever: `"mean"` (1 column), `"mean_spread"` (2), or the full per-level probabilities (5). More columns is more signal and more overfitting room (inferred). - The cookbook's own **Next steps** list, condensed: screen a candidate question *as a state* with nouls (answerable from the text? unambiguous under its criteria? applies to most rows? varies across rows?) before paying to answer it; prune correlated columns by clustering; add TF-IDF / embedding baselines; mix proposer model families and deduplicate before Jev sees them; try non-CatBoost regressors; match validation to deployment (chronological or grouped splits); stop on a plateau of CV RMSE or at a request budget; check stability across seeds. ## Gotchas - **`cooksafe` is not publicly installable.** Install line: `pip install anthropic openai catboost numpy matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a TypeSafe helper on a private index, and `pypi.typesafe.ai` returned 404 publicly on 2026-09-17. Use `pip install typesafe-sdk anthropic openai catboost numpy` and reimplement `JsonCache(Path("json_cache.json"))` — a decorator memoizing JSON-serializable return values to a file keyed by the function name and how each argument was spelled (the cookbook notes it passes `seed=` as a keyword for exactly that reason, and rounds every number entering a prompt so replays hit) — and `make_playground_link(state, questions, models=[...])`. - **A `Score` takes at most ten levels.** Stated here: "Ten because ten levels is the most a `Score` question takes - eleven comes back as a server error." Matches `raw/docs/primitives__score.md` ("at least two levels and takes up to 10"). See [[concepts/score]]. - **Cost scales with rows, not questions.** "The request count grows with rows, not with questions: one request per row per round, so 100,000 rows is 100,000 requests a round. A revision counts as a new question, so it costs another pass over every row." - **Concurrency ceiling.** "Raise the worker pool slowly. Eight is already enough to hit a rate limit on a shared key." See [[reference/rate-limits-and-errors]]. - **Asking Jev for the score directly needs a learned offset.** "nothing in the question says where this publication's scores actually sit on it. So every answer is then moved by a single offset, measured on the dev scores… it is the only thing this shortcut learns from the scores." The offset was `-1.71`. - **`probabilities.get(i, 0.0)`** — Score probabilities are keyed by integer level, and the code does not assume every level is present. - **`NoulCriteria` is both constructed with keywords and read like a mapping** (`PRESENCE_CRITERIA['true']` in a print statement). - **Feature ids are `name@round`** so a revised question does not clobber an earlier round's cached columns; `slug()` forces names to be plain and unique because they become question ids *and* column labels. - **`importance share` is narrow.** "It is not a share of rows, of questions, or of prediction accuracy." - **Model pinning.** Numbers are `jev-1.12` + `claude-sonnet-5`; `jev-latest` now resolves to `jev-1.13.0`. The `gpt-5.6-luna` proposer branch exists in the code but "was not run." ## Related - [[cookbooks/overview]] — the cookbook index - [[concepts/score]] — levels, the 10-level cap, `probabilities` keyed by level - [[concepts/noul]] — `Noul` and `NoulCriteria` - [[patterns/composite-scoring]] — combining several Jev answers into one number - [[patterns/fan-out]] — many questions, one request per row - [[concepts/how-to-build]] — decomposing a judgment into atomic questions - [[guides/writing-instructions-and-criteria]] — what makes a proposed question answerable - [[reference/python-sdk-questions]] — `Score`, `Noul`, `NoulCriteria` signatures ## Sources - raw/docs/cookbooks__autoresearch_feature_discovery.md (https://docs.typesafe.ai/cookbooks/autoresearch_feature_discovery)