--- title: "Cookbook: Parallel questions" type: cookbook tags: [cookbook, batching, cost, latency, fan-out] created: 2026-09-17 updated: 2026-09-17 confidence: high sources: - raw/docs/cookbooks__parallel_questions.md jev_version: "jev-1.13.0" sdk_python: "0.6.0" summary: "Batching 13 questions about one 54k-character document into a single Jev call was 12.2x cheaper and 10.0x faster than 13 single-question calls, with identical answers." --- # Cookbook: Parallel questions > **TL;DR** Every question is scored on its own against the document, so batching N questions into one `system_one` call changes neither the mean nor the run-to-run std dev of any answer. It changes the bill: on a ~54,000-character article, 13 questions in one call cost `$0.000497` in `0.27s` versus `$0.006090` in `2.71s` for 13 separate calls — **12.2x cheaper, 10.0x faster**. ## Goal Settle the question "does putting N questions in one request change the answers?" empirically, and quantify what batching saves. The experiment asks each question 5 times both ways — all N in one request, and one question per request — and compares the run-to-run standard deviation: how far an answer moves from one repeat to the next. The case is a regulatory briefing: a compliance team wants 13 things checked against the GDPR article. ## Inputs / state shape The state is a dict wrapping a pinned Wikipedia revision: ```python WIKIPEDIA_REVISION = 1363040264 # "General Data Protection Regulation", as of 2026-07 DOCUMENT = { "source": f"https://en.wikipedia.org/?oldid={WIKIPEDIA_REVISION}", "text": fetch_article(WIKIPEDIA_REVISION), } ``` and each call sends `state={"article": DOCUMENT}`. The article is **53,777 characters** — a document-dominated workload where the document is most of every request. The revision is pinned and cached so the numbers stay fixed as the live article is edited. Constants: `TYPESAFE_MODEL = "jev-1.12"`, `PRICE = (0.042, 0.00)` ($ per 1M input/output tokens, jev-1.12 as of 2026-09), `RUNS = 5`. ## Questions asked Thirteen questions: 8 `Noul`, 2 `Choice`, 3 `Score`. Verbatim. **Nouls** (`instructions` only, no `criteria`): | key | `instructions` | |---|---| | `breach_72h` | `Must a personal data breach be reported to the supervisory authority within 72 hours?` | | `applies_non_eu` | `Does the regulation apply to organisations established outside the EU that offer goods or services to people in the EU?` | | `dpo_all_orgs` | `Must every organisation appoint a Data Protection Officer, regardless of what data it processes?` | | `pre_ticked_consent` | `Can valid consent be obtained through pre-ticked boxes or inactivity?` | | `right_erasure` | `Does the regulation grant individuals a right to erasure of their personal data?` | | `data_portability` | `Does the regulation include a right to data portability?` | | `us_federal_law` | `Is the GDPR a United States federal law?` | | `criminal_penalties` | `Does the GDPR itself impose criminal penalties such as imprisonment?` | **Choices** (`criteria` maps each label to its meaning): ```python "instrument_type": Choice( instructions="What kind of EU legal instrument is the GDPR?", criteria={ "Regulation": "Directly binding law in all member states, no national implementation needed.", "Directive": "Sets goals that member states implement through national law.", "Treaty": "An international treaty between states.", "Recommendation": "Non-binding guidance.", }, ), "max_fine": Choice( instructions="What is the maximum administrative fine for the most serious infringements?", criteria={ "TwentyM_or_4pct": "Up to EUR 20 million or 4% of annual worldwide turnover, whichever is greater.", "TenM_or_2pct": "Up to EUR 10 million or 2% of annual worldwide turnover, whichever is greater.", "FixedCap": "A fixed amount not tied to turnover.", "NoFines": "The GDPR provides no administrative fines.", }, ), ``` **Scores** (`criteria` lists level descriptions, from level 0 up — already the 0.6.0 ordered-sequence form): ```python "individual_rights": Score( instructions="How strong are the rights the GDPR grants to individuals over their data?", criteria=[ "None: individuals get no rights over their data.", "Weak: a right to be informed, but little control.", "Moderate: access and correction rights, but limited means to act on them.", "Strong: access, erasure, portability, and objection rights, with enforcement behind them.", ], ), "penalty_severity": Score( instructions="How severe are the penalties the GDPR provides for non-compliance?", criteria=[ "None: no penalties of any kind.", "Symbolic: small fixed fines unlikely to change behavior.", "Substantial: fines large enough to matter to most companies.", "Severe: fines scaled to global revenue, material even to the largest companies.", ], ), "compliance_burden": Score( instructions="How heavy is the compliance burden the GDPR places on organisations?", criteria=[ "Negligible: no meaningful obligations.", "Light: a few notices and disclosures.", "Moderate: documented processes and some dedicated roles for larger processors.", "Heavy: records, impact assessments, officers, and breach procedures for many organisations.", "Extreme: obligations so demanding that ordinary organisations cannot fully comply.", ], ), ``` One number is tracked per answer, by type: `Noul` → the probability of "yes"; `Choice` → the max prob, the probability on the picked label; `Score` → the score normalized to 0–1, the score divided by the top level. ## Combining logic in code ```python import os from statistics import mean, stdev from time import perf_counter from typesafe_sdk import Choice, ChoiceAnswer, Noul, NoulAnswer, Score, TypeSafeClient TYPESAFE_MODEL = "jev-1.12" PRICE = (0.042, 0.00) RUNS = 5 client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=120.0) def ask(keys: tuple[str, ...], run: int): """One TypeSafe call -> ({key: tracked metric}, input_tokens, output_tokens, latency_s); ``run`` only forces a distinct live call per repeat.""" started = perf_counter() response = client.system_one( state={"article": DOCUMENT}, questions={key: QUESTIONS[key] for key in keys}, model=TYPESAFE_MODEL, ) values = {} for key in keys: answer = response.answers[key] if isinstance(answer, NoulAnswer): values[key] = answer.noul elif isinstance(answer, ChoiceAnswer): values[key] = max(answer.probabilities.values()) else: values[key] = answer.score / (len(QUESTIONS[key].criteria) - 1) return ( values, response.usage.input_tokens, response.usage.output_tokens, perf_counter() - started, ) def priced(result): """({key: metric}, in_tokens, out_tokens, latency) -> ({key: metric}, cost_usd, latency).""" values, input_tokens, output_tokens, latency = result return values, input_tokens / 1e6 * PRICE[0] + output_tokens / 1e6 * PRICE[1], latency batched = [priced(ask(tuple(QUESTIONS), run)) for run in range(RUNS)] singles = [{key: priced(ask((key,), run)) for key in QUESTIONS} for run in range(RUNS)] ``` Note `answer.score / (len(QUESTIONS[key].criteria) - 1)`: the normalizer is the **top level index**, so a 4-level `Score` divides by 3 and the 5-level `compliance_burden` divides by 4. This is one of the places where the 0.6.0 ordered-sequence `criteria` makes the arithmetic obvious. Cost is applied *after* the cached call, so a price change needs no new API calls. ## Results / what the cookbook reports **Answers (5 runs per strategy).** Mean and std dev of each question's tracked number: | question | metric | batched mean | single mean | batched std | single std | |---|---|---|---|---|---| | `breach_72h` | p(yes) | 0.804 | 0.814 | 0.0055 | 0.0055 | | `applies_non_eu` | p(yes) | 0.990 | 0.990 | 0.0000 | 0.0000 | | `dpo_all_orgs` | p(yes) | 0.030 | 0.030 | 0.0000 | 0.0000 | | `pre_ticked_consent` | p(yes) | 0.040 | 0.040 | 0.0000 | 0.0000 | | `right_erasure` | p(yes) | 0.990 | 0.990 | 0.0000 | 0.0000 | | `data_portability` | p(yes) | 0.990 | 0.990 | 0.0000 | 0.0000 | | `us_federal_law` | p(yes) | 0.010 | 0.010 | 0.0000 | 0.0000 | | `criminal_penalties` | p(yes) | 0.108 | 0.108 | 0.0045 | 0.0084 | | `instrument_type` | max prob | 1.000 | 1.000 | 0.0000 | 0.0000 | | `max_fine` | max prob | 1.000 | 1.000 | 0.0000 | 0.0000 | | `individual_rights` | normalized score | 1.000 | 1.000 | 0.0000 | 0.0000 | | `penalty_severity` | normalized score | 1.000 | 1.000 | 0.0000 | 0.0000 | | `compliance_burden` | normalized score | 0.750 | 0.750 | 0.0000 | 0.0000 | Choices, scores, and six of the eight nouls came back **identical across all 5 repeats** under both strategies. `breach_72h` and `criminal_penalties` carry a little run-to-run sampling noise, the same size either way, with means agreeing to within that noise — the noise is a property of the question, not of how you batch. **Cost and speed:** ``` batching calls cost total time one call, all 13 1 $0.000497 0.27s 13 calls, one each 13 $0.006090 2.71s batching: 12.2x cheaper, 10.0x faster ``` The speed figure **sums the 13 single-call latencies**, so it assumes they run one after another. Fire them concurrently and the gap shrinks, but the 13x token cost stays. The bigger the document, the nearer the saving comes to a full Nx. ## Adapting it to a new domain 1. Put the shared document in `state` once and give every question a distinct key. 2. Keep the per-type metric extraction (`answer.noul`, `max(answer.probabilities.values())`, `answer.score / (levels - 1)`) so heterogeneous questions land on one comparable scale. 3. To audit stability in your own domain, re-run the same batched call `RUNS` times with a `run` argument that only exists to defeat caching, and report per-question std dev. 4. Price after the fact from `response.usage.input_tokens` / `output_tokens` rather than baking a rate into the call path. ## Gotchas - **The saving is document-dominated.** With a short state and many long questions, batching saves much less; the article here is 53,777 characters against 13 short instructions. - **Latency comparison is sequential by construction.** Concurrent single calls narrow the wall-clock gap but not the token bill. - **Not every answer is noise-free.** Two of thirteen questions moved run to run; if you need a stable number near a decision threshold, measure your own question rather than assuming std dev 0. - **Choice normalization.** The tracked `Choice` metric is the max probability, which will read 1.000 on unambiguous questions and tells you nothing about *which* label won — keep `answer.choice` too if you care. - **Install line.** `pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/`. `cooksafe` is a helper package on TypeSafe's private index, and `pypi.typesafe.ai` returned **404 publicly as of 2026-09-17** — install `pip install typesafe-sdk` and reimplement the two helpers: `JsonCache(Path("json_cache.json"))`, a decorator memoizing each call's return value into a JSON file keyed on its arguments (delete to re-run live), and `make_playground_link(state, questions, models=[...])`, which packs the state and questions into a `https://console.typesafe.ai/playground#share/...` URL. - Published run used `jev-1.12`; this wiki documents `jev-1.13.0`. ## Related - [[patterns/fan-out]] — the pattern this cookbook measures - [[concepts/primitives]] — Noul, Choice and Score side by side - [[concepts/score]] — level criteria and score normalization - [[reference/models-and-pricing]] — the token rate behind `PRICE` - [[reference/python-sdk-responses]] — `response.usage`, `NoulAnswer`, `ChoiceAnswer` - [[cookbooks/rerank]] — a cookbook that deliberately asks one question per pair, and says so - [[cookbooks/overview]] — the full cookbook catalog ## Sources - raw/docs/cookbooks__parallel_questions.md (https://docs.typesafe.ai/cookbooks/parallel_questions.md)