$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/concepts

Score questions

[ concept ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#score · primitives · levels · rubric · confidence

TL;DR {"type": "score", "instructions": "...", "criteria": ["level 0 …", "level 1 …", …]} — criteria is an ordered array, at least 2 levels and up to 10. The answer is {"type": "score", "score": <float 0..len-1>, "legend": {...}, "probabilities": {"0": …}, "confidence": 0..1}. score is the probability-weighted mean of the level numbers, so different distributions can produce the same score — always read probabilities and confidence too.

When to use / when not to use

Use a Score when the answer is a position on a spectrum and you can describe what each point on that spectrum means: bug severity, customer frustration, report quality, skill level.

Example Score questions from raw/docs/primitives__score.md:

"How severe is the bug being reported?"
  → 0: Cosmetic; no impact to functionality
  → 1: Broken or degraded feature, but workaround exists
  → 2: Blocking issue; no workaround exists

"How formal is this outfit based on the description"
  → 0: gym clothes
  → 1: casual
  → 2: business casual
  → 3: formal
  → 4: black tie

"How relevant is this candidate's experience to the job posting"
  → 0: completely unrelated
  → 1: adjacent field
  → 2: some direct experience
  → 3: deep, direct experience

Request contract

Same three top-level fields as any other question type: state, model, questions.

Field Type Required Description
type "score" yes Always "score".
instructions string | object | array yes What the model should rate.
criteria array yes An ordered array of level descriptions, from the low end of the scale to the high end. At least two levels, up to 10.

instructions and each level in criteria may be a string, an object, an array, or null (all are EntryType) — see Structured instructions, options, levels, criteria.

Levels are an ordered sequence

Each entry in criteria is a level: one point on the spectrum, described in words. A level's number is its position in the array, starting at 0, so a three-entry array gives levels 0, 1 and 2. The order of the array is the numbering. In SDK 0.6.0 Score.criteria is an ordered sequence, not an int-keyed dict.

The model gets the descriptions and nothing else. Each level is judged on its own against the state — it does not see a level's number or its neighbours.

The score in the response is a position on that spectrum: for a three-level scale it runs from 0 to 2, and it can land between two levels.

Minimal request

{
  "state": "The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari.",
  "model": "jev-latest",
  "questions": {
    "bug_severity": {
      "type": "score",
      "instructions": "How severe is the reported issue?",
      "criteria": [
        "Cosmetic; no impact to functionality",
        "Broken or degraded feature, but workaround exists",
        "Blocking issue; no workaround exists"
      ]
    }
  }
}

from typesafe_sdk import Score, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari.",
        questions={
            "bug_severity": Score(
                instructions="How severe is the reported issue?",
                criteria=[
                    "Cosmetic; no impact to functionality",
                    "Broken or degraded feature, but workaround exists",
                    "Blocking issue; no workaround exists",
                ],
            ),
        },
    )

    print(response.answers["bug_severity"].score)

The question id (bug_severity) is not sent to the model; the answer comes back under the same id. In the JavaScript SDK the helper is score(instructions, criteria) with "At least two descriptions indexed by score from zero; entries may be null" — see JavaScript/TypeScript SDK: install, client, choice/score/noul.

Response contract

Field Type Description
type "score" Matches the question type.
score number The position on the level number line, from 0 to the top level number. Each level number multiplied by its probability, added up. Can land between levels.
legend map<string, string> Each level number mapped back to its description.
probabilities map<string, number> The probability of each level, keyed by level number as a string. The values sum to 1.
confidence number 0 to 1, computed from how probabilities is spread. A single peak means high confidence; probability spread over several levels means low confidence. 
{
  "model": "jev-latest",
  "answers": {
    "bug_severity": {
      "type": "score",
      "score": 1.3,
      "confidence": 0.54,
      "legend": {
        "0": "Cosmetic; no impact to functionality",
        "1": "Broken or degraded feature, but workaround exists",
        "2": "Blocking issue; no workaround exists"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.7,
        "2": 0.3
      }
    }
  },
  "usage": {
    "input_tokens": 332,
    "output_tokens": 18
  }
}

score here is 0 × 0.0 + 1 × 0.70 + 2 × 0.30 = 1.30. That matches the report: the export is broken, and switching to Chrome is a workaround for most customers but not for the ones who only use Safari. Confidence is 0.54 because the distribution is split.

SDK note: using the Python SDK, ScoreAnswer has score, confidence, probabilities, and legend as typed fields, and it keys probabilities and legend by integer level rather than by string. See Python SDK responses, answers, usage, models.

Reading a Score

Same question and levels, different bug reports:

State score confidence Level 0 Level 1 Level 2
The export button is misaligned by a few pixels on the settings page. 0.0 1.0 1.0 0.0 0.0
The PDF export button does nothing when clicked. I can still export to CSV and convert it myself, but that takes ages. 1.0 1.0 0.0 1.0 0.0
Export to PDF fails with a spinner that never finishes. Some of our team say CSV export still works for them, others say it fails too. 1.12 0.81 0.0 0.88 0.12
The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari. 1.3 0.54 0.0 0.7 0.3
Nobody on our team can log in since this morning. We get a 500 error on every attempt. 2.0 1.0 0.0 0.0 1.0

(Levels 0/1/2 columns are probabilities.)

Key readings:

Writing good levels

instructions: "Rate severity from 0 to 2, where 2 is worst"
criteria: ["0", "1", "2"]
→ score 0.57, confidence 0.35, probabilities 0: 0.43, 1: 0.57, 2: 0.0

The same report with the three descriptive levels scores 0.0 at confidence 1.0.

Splitting a complex judgment into several Scores

A judgment that depends on several things is best split into one Score per thing, then combined in code with weights for relative importance. The weights are yours: when the combined result doesn't match what your team would decide, change them in code and run again. Send all the Score questions in one request — they run in parallel.

Request (the spinner ticket with more context):

{
  "state": "Export to PDF fails with a spinner that never finishes. Some of our team say CSV export still works for them, others say it fails too. This is the third time I'm writing in and honestly I'm done. Steps: open any report, click Export, choose PDF. Chrome 128 on macOS.",
  "model": "jev-latest",
  "questions": {
    "severity": {
      "type": "score",
      "instructions": "How severe is the reported issue?",
      "criteria": [
        "Cosmetic; no impact to functionality",
        "Broken or degraded feature, but workaround exists",
        "Blocking issue; no workaround exists"
      ]
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": [
        "Calm, just stating facts",
        "Frustrated but civil",
        "Very angry, strong language or threatening to leave"
      ]
    },
    "report_quality": {
      "type": "score",
      "instructions": "How much does the report give an engineer to work with?",
      "criteria": [
        "No detail; just says something is broken",
        "Names the feature but no steps or environment",
        "Steps to reproduce or environment, but not both",
        "Steps to reproduce and environment"
      ]
    }
  }
}

Response:

{
  "model": "jev-latest",
  "answers": {
    "severity": {
      "type": "score",
      "score": 1.24,
      "confidence": 0.63,
      "legend": {
        "0": "Cosmetic; no impact to functionality",
        "1": "Broken or degraded feature, but workaround exists",
        "2": "Blocking issue; no workaround exists"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.76,
        "2": 0.24
      }
    },
    "frustration": {
      "type": "score",
      "score": 1.45,
      "confidence": 0.33,
      "legend": {
        "0": "Calm, just stating facts",
        "1": "Frustrated but civil",
        "2": "Very angry, strong language or threatening to leave"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.55,
        "2": 0.45
      }
    },
    "report_quality": {
      "type": "score",
      "score": 3.0,
      "confidence": 1.0,
      "legend": {
        "0": "No detail; just says something is broken",
        "1": "Names the feature but no steps or environment",
        "2": "Steps to reproduce or environment, but not both",
        "3": "Steps to reproduce and environment"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.0,
        "2": 0.0,
        "3": 1.0
      }
    }
  },
  "usage": {
    "input_tokens": 468,
    "output_tokens": 43
  }
}

Normalize before combining

The three scales have different lengths: a four-level scale returns 0 to 3 and a three-level scale returns 0 to 2, so a top score on one is bigger than a top score on the other. Divide each score by its top level number, len(criteria) - 1, to put every score on 0 to 1. Then the weights mean what they say.

from typesafe_sdk import Score, TypeSafeClient

TRIAGE_QUESTIONS = {
    "severity": Score(
        instructions="How severe is the reported issue?",
        criteria=[
            "Cosmetic; no impact to functionality",
            "Broken or degraded feature, but workaround exists",
            "Blocking issue; no workaround exists",
        ],
    ),
    "frustration": Score(
        instructions="How frustrated is the customer?",
        criteria=[
            "Calm, just stating facts",
            "Frustrated but civil",
            "Very angry, strong language or threatening to leave",
        ],
    ),
    "report_quality": Score(
        instructions="How much does the report give an engineer to work with?",
        criteria=[
            "No detail; just says something is broken",
            "Names the feature but no steps or environment",
            "Steps to reproduce or environment, but not both",
            "Steps to reproduce and environment",
        ],
    ),
}


def normalized(answers, question_id: str) -> float:
    """Put a score on 0 to 1 by dividing by its top level number."""
    top_level = len(TRIAGE_QUESTIONS[question_id].criteria) - 1
    return answers[question_id].score / top_level


def priority(ticket: str) -> float:
    with TypeSafeClient() as client:
        response = client.system_one(
            state=ticket,
            questions=TRIAGE_QUESTIONS,
        )
    answers = response.answers

    severity = normalized(answers, "severity")
    frustration = normalized(answers, "frustration")
    report_quality = normalized(answers, "report_quality")

    # A detailed report helps an engineer investigate, so it raises priority a little.
    return 0.6 * severity + 0.3 * frustration + 0.1 * report_quality

For the response above the normalized scores are 0.62 (severity), 0.725 (frustration), and 1.0 (report quality), so the priority is 0.6 × 0.62 + 0.3 × 0.725 + 0.1 × 1.0 = 0.6895, which rounds to 0.69. This is the Composite scoring pattern.

Structured level descriptions

Start with plain strings. When the model keeps scoring between two neighbouring levels on inputs you think are clear, give each level an object with a field for what the level covers and a field with a few example situations. Use the same field names on every level so the model can compare like with like.

{
  "state": "Export to PDF fails with a spinner that never finishes. Some of our team say CSV export still works for them, others say it fails too.",
  "model": "jev-latest",
  "questions": {
    "bug_severity": {
      "type": "score",
      "instructions": "How severe is the reported issue?",
      "criteria": [
        {
          "what": "Cosmetic; no impact to functionality",
          "examples": ["typo in a label", "misaligned icon"]
        },
        {
          "what": "Broken or degraded feature, but workaround exists",
          "examples": ["export fails in one browser but works in another"]
        },
        {
          "what": "Blocking issue; no workaround exists",
          "examples": ["cannot log in", "data loss"]
        }
      ]
    }
  }
}

The legend echoes the structured levels back:

{
  "model": "jev-latest",
  "answers": {
    "bug_severity": {
      "type": "score",
      "score": 1.06,
      "confidence": 0.91,
      "legend": {
        "0": {
          "what": "Cosmetic; no impact to functionality",
          "examples": [
            "typo in a label",
            "misaligned icon"
          ]
        },
        "1": {
          "what": "Broken or degraded feature, but workaround exists",
          "examples": [
            "export fails in one browser but works in another"
          ]
        },
        "2": {
          "what": "Blocking issue; no workaround exists",
          "examples": [
            "cannot log in",
            "data loss"
          ]
        }
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.94,
        "2": 0.06
      }
    }
  },
  "usage": {
    "input_tokens": 379,
    "output_tokens": 18
  }
}

With plain strings this ticket scored 1.12 at confidence 0.81; with examples it scores 1.06 at 0.91.

Examples only help when they look like your real inputs. The opening Safari report with three different sets of level objects:

Level description score confidence
plain string: no object with examples 1.30 0.54
Added examples array with useful example: "export fails in one browser but works in another" 1.07 0.90
Added examples array with example unrelated to browsers: "search fails, but browsing categories still works" 1.28 0.57

The matching example concentrates more probability on one level; the unrelated example changes the result only slightly compared with plain strings. Higher confidence does not establish which answer is correct. Choose examples with known expected levels, then test the revised descriptions on separate inputs before keeping them.

Gotchas

Related

Sources