---
title: "Speculative fan-out"
type: pattern
tags: [patterns, fan-out, speculative, latency, cost]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/patterns__fan-out.md
  - raw/docs/patterns.md
  - raw/github/skills/skills/typesafe-ai/SKILL.md
jev_version: "jev-1.13.0"
summary: "Put every question your decision tree could need into one call, including speculative ones, and let code discard the irrelevant answers."
---

# Speculative fan-out

> **TL;DR** Send all the questions your system could need in a single `POST /v1/systemone` call — including ones that only matter on branches you have not taken yet — then branch in code. "All questions are evaluated in parallel, so adding more questions to a call typically doesn't add any latency to the response."

## Problem

A decision tree has dependent branches: you classify a support ticket, and *only if* it is a bug report do you need its severity and whether it has reproduction steps; *only if* it is billing do you need to know whether a refund was requested.

The obvious implementation asks the classifying question first, waits, then issues a second call for the branch-specific questions. That serializes two round trips for what is logically one decision, and the second call re-sends the same state.

## Pattern

From `raw/docs/patterns__fan-out.md`:

> Because TypeSafe supports sending many questions in a single API call, we recommend putting all of the questions your system needs in a single request, and then using code to decide what is relevant after the fact. All questions are evaluated in parallel, so adding more questions to a call typically doesn't add any latency to the response.

A **speculative question** is one you ask before you know whether its answer will be used. The source's definition:

> **Speculative questions:** `bug_severity` and `has_reproducible_steps` only matter if the ticket is a bug report. `refund_requested` only matters for billing. We include all upfront because there is no speed cost for additional questions. If the ticket turns out to be a feature request, the bug severity result will be irrelevant, in which case your code path simply ignores it.

Two steps: fan out, then route with code.

## Implementation

### Step 1: speculative fan-out

The documented example is support ticket triage. State and questions, verbatim from the source (the source renders these as a playground example; `state` and `questions` are the two top-level fields of the request body — add `"model": "jev-latest"` to make it a complete HTTP request, see [[reference/http-api]]):

```json title="request"
{
  "state": "Hi, I placed an order (#98423) last Thursday and was charged twice. I also can't log in after the site update, and adding Apple Pay would be really helpful. This is getting frustrating.",
  "questions": {
    "category": {
      "type": "choice",
      "instructions": "Determine the broad category of this support ticket",
      "criteria": {
        "bug_report": "The user is reporting something that is broken or producing errors",
        "billing": "Charges, invoices, refunds, subscriptions",
        "feature_request": "The user is requesting new functionality",
        "account": "Login, permissions, profile, security"
      }
    },
    "bug_severity": {
      "type": "score",
      "instructions": "How severe is the reported issue",
      "criteria": [
        "Cosmetic; no impact to functionality",
        "Broken or degraded feature; workaround exists",
        "Blocking issue; no workaround exists"
      ]
    },
    "has_reproducible_steps": {
      "type": "noul",
      "instructions": "The user describes specific steps to reproduce the issue"
    },
    "refund_requested": {
      "type": "noul",
      "instructions": "The user is explicitly asking for a refund or credit"
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated the user appears",
      "criteria": ["Calm, matter-of-fact", "Frustrated but civil", "Very angry"]
    }
  }
}
```

Note the mix: one Choice, two Scores, two Nouls — all in one call. See [[concepts/choice]], [[concepts/score]], [[concepts/noul]].

### Step 2: route with code

> Your code decides what is relevant based on the classification result:

```python title="triage.py"
category = response.answers["category"]
bug_severity = response.answers["bug_severity"]
bug_repro = response.answers["has_reproducible_steps"]
refund = response.answers["refund_requested"]
frustration = response.answers["frustration"]

if category.choice == "bug_report":
    if bug_severity.score > 1.5 and bug_repro.noul > 0.6:
        escalate_to_engineering(ticket_id, severity="high")
    else:
        add_to_bug_backlog(ticket_id)

elif category.choice == "billing":
    if refund.noul > 0.7:
        route_to_billing_with_flag(ticket_id, refund_likely=True)
    else:
        route_to_billing(ticket_id)

elif category.choice == "feature_request":
    log_feature_request(ticket_id)

# Frustration is useful regardless of category
if frustration.score > 1.5:
    flag_for_priority_response(ticket_id)
```

The source's closing point: "Everything needed for the full decision tree comes from one call. Speculative questions are ignored when irrelevant and save a round trip when they are not."

Note that `frustration` is not speculative — it is consumed on every branch. Fan-out mixes both kinds freely.

### Sending it

The example above is JSON for `POST /v1/systemone`. To run the same fan-out through an SDK, build the same `questions` map with `Choice` / `Score` / `Noul` (Python) or `choice()` / `score()` / `noul()` (JS) — see [[guides/quickstart]] for a complete runnable call, and [[reference/python-sdk]] / [[reference/javascript-sdk]] for signatures.

## When it fails

- **The next question depends on the previous *answer*, not just a branch.** The agent skill draws the line: "A second request is warranted when an earlier answer is needed to fetch evidence, construct new state, or determine the next options." Fan-out only collapses trees whose questions can all be written against the state you already have.
- **A speculative question with an unstated premise.** `bug_severity` is asked of tickets that may not describe a bug. The skill's rule: "State each speculative premise explicitly; code consumes the applicable answers." A question that silently assumes its branch produces meaningless numbers on the other branches — which is fine while you ignore them, and a bug the moment you do not.
- **Tokens are not free even though latency is.** "Adding more questions to a call typically doesn't add any latency" is a latency claim, not a cost claim. The skill: "Extra questions still use tokens; measure actual request budgets, cost, and end-to-end latency." The `usage` object reports `input_tokens` and `output_tokens` per call; see [[reference/models-and-pricing]].
- **Reading a speculative answer's confidence as a system-health signal.** Low confidence on an unused branch means nothing. The skill: "Ignore uncertainty on unused branches."
- **Very wide fan-outs.** The pattern page gives no ceiling on question count; check [[reference/models-and-pricing]] for the context window and [[reference/rate-limits-and-errors]] before fanning out to dozens of questions (inferred — the source states no limit).

## Variants

- **Fan-out plus a confidence gate.** [[patterns/intent-routing]] asks `intent` and `complexity` together and gates on `intent.confidence` before consuming either.
- **Fan-out plus an LLM fallback.** [[guides/smart-home-demo]] fans out a long question list per utterance and hands the request to an LLM only when the classification says the user wants conversation.
- **Fan-out as typed argument filling.** The agent skill's "Route and fill known arguments" direction: "A request can select a handler and its typed parameters. Ask useful branch-specific questions up front and consume only the relevant answers." See [[cookbooks/function-calling]].
- **Fan-out over independent items.** The same call shape applied per document, in parallel; see [[cookbooks/parallel-questions]].

## Related

- [[patterns/overview]] — the four-pattern catalog
- [[patterns/intent-routing]] — classification plus routing, built on this pattern
- [[concepts/state]] — the payload every fanned-out question shares
- [[concepts/how-to-build]] — decomposing a workflow into atomic questions
- [[reference/http-api]] — the `questions` map on the wire
- [[guides/smart-home-demo]] — fan-out in a running demo

## Sources

- raw/docs/patterns__fan-out.md (https://docs.typesafe.ai/patterns/fan-out)
- raw/docs/patterns.md (https://docs.typesafe.ai/patterns)
- raw/github/skills/skills/typesafe-ai/SKILL.md (https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md)