---
title: "Confidence vs probability"
type: concept
tags: [confidence, probabilities, thresholds, routing, calibration]
created: 2026-09-17
updated: 2026-09-17
confidence: high
sources:
  - raw/docs/confidence.md
  - raw/docs/introduction.md
  - raw/docs/concepts__how-to-build-with-system-one.md
  - raw/docs/models.md
jev_version: "jev-1.13.0"
sdk_python: "0.6.0"
summary: "confidence is a 0-1 statistic derived from an answer's probabilities distribution; Choice and Score carry it, Noul does not."
---

# Confidence vs probability

> **TL;DR** `probabilities` is the full distribution over your Choice options or Score levels. `confidence` collapses that distribution's *shape* into one number from 0 to 1 so you can threshold on it directly. Choice and Score answers carry `confidence`; **Noul answers do not**. Gate different actions at different thresholds according to their stakes.

## The two fields

| Field | On which answers | Type | Meaning |
|---|---|---|---|
| `probabilities` | Choice, Score | distribution over options (Choice) or levels (Score) | The model's probability mass on each possible outcome. |
| `confidence` | Choice, Score | number, 0 to 1 | A statistic computed from `probabilities` that summarizes how concentrated the distribution is. |
| `choice` | Choice | the option key | The selected option. |
| `score` | Score | number | The score on your rubric. |
| `noul` | Noul | number, 0–1 | The probability the statement is true. Noul answers do not carry a `confidence` property. |

(The `choice` / `score` / `noul` rows and the "Noul (0–1)" typing come from `raw/docs/introduction.md`; the rest from `raw/docs/confidence.md`.)

## How confidence differs from probability

All Score and Choice answers include a `probabilities` property representing the probability distribution across the options (for Choice) or levels (for Score). **The shape of that distribution is what tells you how certain the model is**: concentrated on one outcome means a confident answer, spread out means an uncertain one.

`confidence` is a statistic computed from the probability distribution the answer already gives you. TypeSafe computes it for you and returns it on every Choice and Score answer, so the common case needs no extra work on your side.

The distinction that matters in code:

- A **probability** answers "how likely is *this particular outcome*?"
- **Confidence** answers "how decisive is the distribution as a whole?"

A flatter distribution means lower confidence. Low confidence on a Choice often means **none of the options are a clear winner** over the others. Low confidence on a Score often means the levels are **ambiguous, multi-dimensional, or the state doesn't contain enough to go on** — which is usually a signal to fix the question or the state, not just to escalate.

> **A solid default.** TypeSafe provides `confidence` "as a convenient measure that fits most use-cases, but you are never locked into our definition. Depending on what you are evaluating, a different measure may serve you better, which is exactly why we give you the full `probabilities` in the response." The docs do **not** publish the exact formula; they defer the pros and cons of different computations to a future cookbook. If you need a specific measure (entropy, margin between top-two, etc.), compute it yourself from `probabilities`. (The formula being unpublished is stated in the source; treating margin/entropy as your alternatives is inferred.)

## "I don't know" is a useful signal

> "If an intelligent system, whether human or machine, cannot express honest uncertainty, the system cannot be trusted."

Confidence gives you a built-in mechanism for the model to say "I'm not sure about this one." This lets your code implement different behavior for different levels of certainty, which the docs call the foundation for building systems you can actually rely on.

## Three paths for using confidence

A useful starting pattern is to divide confidence into three ranges, each producing a different system behavior:

| Band | Behavior |
|---|---|
| **High confidence** | Act automatically. The model has a clear read and you can proceed without human involvement. |
| **Medium confidence** | Proceed with caution. The model has a reasonable answer but is not certain. Depending on context, ask the user to confirm, flag for review, or gather more information before acting. |
| **Low confidence** | Do not act. Route to a human, request clarification, or fall back to a different system. The model is telling you it does not have enough information or the question is not a good fit. |

Where you draw those boundaries depends on the stakes. The source does **not** attach numbers to these three bands.

## Thresholds scale with risk

A confidence threshold is not one number. Different actions within the same system should be gated at different levels depending on the consequences of getting it wrong.

```python theme={null}
response = client.system_one(
    state=user_message,
    questions={
        "action": Choice(
            instructions="What is the user trying to do?",
            criteria={
                "check_balance": "View account balance",
                "approve_transfer": "Approve the pending withdrawal request",
                "support": "Get help with an issue",
            },
        ),
    },
)

action = response.answers["action"]
confidence = action.confidence

if confidence < 0.5:
    # Model is genuinely unsure. Don't guess.
    route_to_human(user_message)

elif action.choice == "check_balance":
    # Low stakes. Showing the wrong screen is recoverable.
    show_balance(account_id)

elif action.choice == "approve_transfer":
    if confidence > 0.9:
        # High stakes, high confidence. Proceed with confirmation.
        confirm_then_execute(account_id)
    else:
        # High stakes, moderate confidence. Verify first.
        ask_user_to_confirm(account_id)
```

The `0.5` confidence floor catches anything the model reports as genuinely uncertain. Above that, the threshold for acting without confirmation is higher for a destructive operation (`> 0.9`) than for a read-only one (no extra gate). **Your code encodes the risk tolerance.**

> The correct threshold values depend on your domain and the performance of the model for your use case. Start with conservative thresholds, test with your own data, and adjust as you observe results.

Note the shape of this example: the confidence floor is checked **first**, before the `choice` is branched on. Reading `.choice` without checking `.confidence` throws away the reason to use a calibrated model at all.

Other threshold values that appear elsewhere in the docs, for calibration of your own expectations — not as recommended defaults:

| Threshold | Context | Source |
|---|---|---|
| `confidence < 0.5` | Route to human, any action | raw/docs/confidence.md |
| `confidence > 0.9` | Act on a high-stakes, destructive action | raw/docs/confidence.md |
| `confidence < 0.8` | Route a Choice to human review | raw/docs/concepts__how-to-build-with-system-one.md |
| `confidence < 0.75` | Route a support-ticket topic Choice to human review | raw/docs/concepts__how-to-build-with-system-one.md |
| `confidence >= 0.7` | Trust a Score before acting on its value | raw/docs/concepts__how-to-build-with-system-one.md |

## Gotchas

- **Noul has no `confidence`.** For a yes/no question the probability *is* the uncertainty: `noul` near `0.5` is the uncertain case. Do not write `answers["x"].confidence` against a Noul answer.
- **Confidence is not accuracy for one item.** Calibration is a property of groups of predictions; see [[concepts/machine-learning-primer]].
- **High confidence on the wrong question is still wrong.** A confidently answered broad question hides its compound judgment — decompose first, per [[concepts/how-to-build]].
- **Aliases move thresholds.** If you have tuned confidence thresholds against a specific version, pin that versioned model ID (e.g. `jev-1.13.0`) instead of `jev-latest`. See [[reference/models-and-pricing]].
- **Test thresholds empirically.** The docs suggest plotting confidence against accuracy on your own data rather than adopting any number from the docs.

## Related

- [[concepts/primitives]] — which answers carry which fields
- [[concepts/choice]] / [[concepts/score]] / [[concepts/noul]] — per-primitive response shape
- [[patterns/confidence-routing]] — the pattern this page underpins
- [[concepts/machine-learning-primer]] — why the probabilities are calibrated
- [[concepts/how-to-build]] — where confidence sits in the design workflow
- [[cookbooks/classification-using-confidence]] — a worked recipe
- [[reference/models-and-pricing]] — pinning versions before tuning thresholds

## Sources

- raw/docs/confidence.md (https://docs.typesafe.ai/confidence)
- raw/docs/introduction.md (https://docs.typesafe.ai/introduction) — per-type return fields
- raw/docs/concepts__how-to-build-with-system-one.md (https://docs.typesafe.ai/concepts/how-to-build-with-system-one) — additional threshold examples
</content>
</invoke>