Writing instructions and criteria that Jev reads correctly

TL;DR jev-1.13 answers the question you wrote, not the one you meant. State the exact condition in instructions, put boundary cases in criteria, describe situations rather than degrees, keep true meaning yes, name the parts of the state you mean with backticked paths, and move counting, arithmetic and date comparison into your code. When a wrong answer makes you say "but I meant…", that sentence is the missing half of the instruction.

The one rule

From Jev 1.13 jaggedness: known failure modes (failure mode 1, literal reading):

state the exact condition in the instructions. Be specific. Put boundary cases in the criteria. When you look at a wrong answer and find yourself explaining what you really meant, that explanation is the missing half of the instruction.

Everything below is an application of that rule.

Step 1 — write the instruction

instructions is the question you are asking about the state. This is where your evaluation logic goes. Write it as a clear, specific question, or as a statement for the model to judge.

Checklist:

• One judgment. Not "analyze this message and determine the best course of action" — that needs slow reasoning and is a signal to split (Choosing between Choice, Score, Noul).
• Don't lean on the question id. IDs are for your code; per raw/docs/api.md the key "is not sent to the underlying model and is not used in inference". Write the complete question even when the ID seems self-explanatory.
• Name the part of the state you mean, with a dot-and-index path including the backticks: "Does `ticket.messages[0].text` request a refund?"
• No indirection. A question about a property of a property costs accuracy; so do double negatives.
• Readable by an average person. Upstream's own standard: "Aim for instructions which are easy for the average person to read and understand."
• No arithmetic, counting, or date comparison in the question at all.

Before / after: vagueness

Before / after: pointing at the state

Both after-forms are verbatim from raw/docs/primitives.md.

Step 2 — write the criteria for the type you chose

Choice: descriptions that separate options

The option names and their descriptions are both sent to the model, so write descriptions that separate the options from each other. Use null when the option name is clear on its own ({"calm": None, "frustrated": None, "angry": None}).

Rules:

• Give the full list, not a shortlist — up to 255 options, a few tokens each.
• Add an other or none of the above option when the list might not cover every input.
• When two options are easy to confuse, escalate the description from a string to an object saying what the option covers, what belongs to a neighbouring option instead, and a few example inputs.

Before — two options a ticket can plausibly match:

"criteria": {
  "return_policy": "Returns and refunds",
  "return_status": "Returns and refunds"
}

After — from raw/docs/primitives__choice.md, each option says what it is not for:

{
  "state": "I sent the shoes back a week ago. When do I get my money?",
  "model": "jev-latest",
  "questions": {
    "return_topic": {
      "type": "choice",
      "instructions": {
        "question": "Which returns topic is the customer asking about?",
        "focus": "Classify the information the customer wants."
      },
      "criteria": {
        "return_policy": {
          "what": "Whether and how an item can be returned",
          "not_for": "Progress of a return already sent",
          "examples": [
            "Can I return shoes I've worn once?",
            "How long do I have to return an order?"
          ]
        },
        "return_status": {
          "what": "Progress of a return already sent",
          "not_for": "Whether and how an item can be returned",
          "examples": [
            "Has my return arrived yet?",
            "When will my refund be paid?"
          ]
        }
      }
    }
  }
}

Result: return_status at confidence 1.0. The field names question, focus, what, not_for, examples are not part of the API and none are reserved — you choose them, and the model sees the names along with the values, so use short names that label what follows. More shapes in Structured instructions, options, levels, criteria.

Score: describe situations, not degrees

criteria is an ordered array from the low end to the high end; a level's number is its index, starting at 0. The model gets the descriptions and nothing else, and each level is judged on its own — it never sees a level's number or its neighbours.

Before (raw/docs/primitives__score.md, on a "button misaligned by a few pixels" report):

instructions: "Rate severity from 0 to 2, where 2 is worst"
criteria: ["0", "1", "2"]
→ score 0.57, confidence 0.35, probabilities 0: 0.43, 1: 0.57, 2: 0.0

After — the same report with descriptive levels scores 0.0 at confidence 1.0:

"criteria": [
  "Cosmetic; no impact to functionality",
  "Broken or degraded feature, but workaround exists",
  "Blocking issue; no workaround exists"
]

Rules that follow from this:

• "Worse than the previous level" means nothing to the model; neither do numbers in the descriptions or the instructions.
• Use as many levels as you can describe distinctly, up to 10. Three is fine. Don't add levels you can't describe distinctly.
• Keep each Score to one dimension. A level that says "punctual and smart and experienced" measures three things, and an input high on one and low on another can't be placed: confidence drops and the score means less.
• Give a rare extreme its own level. A sentiment scale ending at "very angry" can add "abusive or threatening"; without it, both messages may score near the top and the score alone may not distinguish them.

Escalating a level to an object. When the model keeps scoring between two neighbouring levels on inputs you think are clear, add examples — using the same field names on every level:

"criteria": [
  {
    "what": "Cosmetic; no impact to functionality",
    "examples": ["typo in a label", "misaligned icon"]
  },
  {
    "what": "Broken or degraded feature, but workaround exists",
    "examples": ["export fails in one browser but works in another"]
  },
  {
    "what": "Blocking issue; no workaround exists",
    "examples": ["cannot log in", "data loss"]
  }
]

On the spinner ticket: plain strings gave 1.12 at confidence 0.81; with examples, 1.06 at 0.91.

But examples only help when they look like your real inputs. The Safari report, three variants:

Higher confidence does not establish which answer is correct. Choose examples with known expected levels, then test the revised descriptions on separate inputs before keeping them.

Noul: keep true meaning yes

criteria is optional: {"true": "...", "false": "..."}, where true is "What a yes (value near 1) means" and false is "What a no (value near 0) means". Phrase the instruction so a high probability means "yes".

Before — a subtle boundary with no criteria:

{"type": "noul", "instructions": "Has the customer contacted support about this before?"}

After — the boundary pinned down:

{
  "type": "noul",
  "instructions": "Has the customer contacted support about this before?",
  "criteria": {
    "true": "Mentions a prior attempt, ticket, or that they have asked before",
    "false": "No sign of any previous contact"
  }
}

Structured after — definition plus examples on each side, from raw/docs/primitives__advanced.md:

{
  "type": "noul",
  "instructions": {
    "question": "Does the `message` ask the recipient to disclose a sensitive credential?",
    "inspect": "message",
    "focus": "Look for a request to send the credential itself, not a request to change or reset it."
  },
  "criteria": {
    "true": {
      "what": "Asks the recipient to reply with, type, or send a password, PIN, one-time code, or other security sensitive answer",
      "examples": [
        "Reply with your password",
        "Send us the 6-digit code you just received"
      ]
    },
    "false": {
      "what": "No sensitive credential is requested",
      "examples": [
        "Reset your password from the settings page",
        "Your statement is ready"
      ]
    }
  }
}

Try your Noul prompts with and without criteria to see which works better in your use case; the instruction alone is enough for most.

Step 3 — avoid negation traps

Three separate findings from the sources:

1. Contradictory criteria. "A Noul where true maps to no and false maps to yes will perform worse." Treat the criteria as an extension of the instruction and align the two.
2. Double negatives are indirection. "Instructions carrying double negatives or complex indirection are answered less reliably." Rewrite "Is it not the case that the customer declined the refund?" as "Did the customer accept the refund?" (inferred rewrite; the source states the rule, not this example.)
3. A question and its negation are not complements. Two Nouls on "I was charged twice for the same order. Can someone look into this?":

So do not implement "not X" by asking X and subtracting, and do not ask both and expect them to agree. Word the one question you actually want.

Likewise, a Noul and a yes/no Choice on the same text can disagree sharply — on "I'm not happy with the fit. What are my options here?": Noul noul 0.22 versus Choice probabilities["yes"] 0.01 at confidence 0.97. Don't carry a threshold tuned on a Noul over to a Choice.

Step 4 — keep numbers and dates out of the prompt

Counting

Ask one question per item and add up the thresholded answers in code:

from typesafe_sdk import Noul, TypeSafeClient

client = TypeSafeClient(model="jev-1.13")
YES = 0.5  # up to you on what you want the threshold to be, depends on your usecase.

items = ["typesafe", "apple", "california", "banana", "likes", "calibration", "orange", "vertex"]

result = client.system_one(
    {"items": items},
    {
        f"item_{i}": Noul(instructions=f"Is `items[{i}]` the name of a fruit?")
        for i in range(len(items))
    },
)

count = sum(result.nouls[f"item_{i}"].noul > YES for i in range(len(items)))

Before asking a counting question at all, ask why the count needs a model: if the unit is something a regular expression or a parser can find, the count belongs in code.

Numeric representations

Jev performs better on semantic representations than numeric ones — English color names beat hex values, high-level languages beat assembly or binary. Do the conversion in code and pass in either the computed number or a named bucket. Keep the model for the part that is genuinely a judgment, such as whether a color reads as a warning.

Where a magnitude is the judgment, bucket it into Score levels rather than asking for a number:

"criteria": [
  "Under $1,000",
  "$1,000 to $10,000",
  "$10,000 to $100,000",
  "$100,000 to $1,000,000",
  "Over $1,000,000"
]

And never reconstruct an exact number by interpolating between two levels: jev-1.13's score levels are weak in numerical calibration. Use the expectation to check a threshold only.

Dates

jev-1.13 reads dates as text, not as ordered quantities. Asking which of two dates comes first, how far apart they are, or whether one falls inside a window is unreliable — worse with mixed formats, relative references, and domain boundaries such as quarters, settlement windows and accrual periods.

Split the work: extraction is a judgment, so give it to the model; arithmetic is not, so keep it in code. Every part of a date is a small closed set — twelve months, thirty-one possible days, a bounded range of years — so extraction becomes a Choice over enumerated options, with an explicit "not stated" option so a missing part is reported rather than guessed. Code assembles the parts into a real date and owns ordering, duration, offset and weekday. Worked version, including relative dates and confidence gating: Cookbook: Date extraction.

Generated values

When the answer space is bounded, turn extraction into a Choice over the options rather than asking for the value itself. For open-ended extraction, extract candidate options using regex or a generative model and let Jev pick the correct one — the shape used in the structured-instructions example in Structured instructions, options, levels, criteria.

Step 5 — shrink the state

Accuracy falls as the state grows with content unrelated to the decision: unrelated detail acts as a distractor, and a large state makes it harder to tell which part of the input produced a wrong answer. Retrieve and filter in code first and send only the fields the question needs. When you can't filter in code, use a Noul to filter for relevance (Cookbook: Classifying RAG passages). See State: what you send Jev.

Note the asymmetry: extra questions are cheap, extra state is not. Adding questions barely changes response time and costs only their tokens; adding irrelevant state costs accuracy.

Step 6 — test edge cases before deploying

• Test the wording on your own data. "Two wordings of the same scale can behave differently on your data."
• Judge by known-correct examples, not by confidence. "Higher confidence alone does not show that a description is better" and confidence 1.0 "describes the model's answer, not a guarantee that the answer is correct."
• Hold out the inputs you tuned on. Choose examples with known expected levels, then test the revised descriptions on separate inputs before keeping them.
• Test adversarial inputs. State is data and jev-1.13 does not treat it as hostile by default; injected instructions, misleading framing, or text that argues for its own classification can move the answer. Be explicit in the criteria and test the integration thoroughly before deploying it to many users.
• Check structural assumptions you are relying on — a Noul threshold, a Choice/Noul equivalence, a sum over negations — because the model does not guarantee them.

See Testing and evaluating a Jev workflow and the consistency cookbooks.

Iteration loop

1. Start with plain strings for instructions and every criteria entry.
2. Run your known-answer set.
3. For each wrong answer, write the sentence that begins "but I meant…". That sentence goes into instructions, or into the criteria as a boundary case.
4. If two options or two neighbouring levels keep getting confused, escalate those entries from strings to objects (what / not_for / examples, or what / examples), consistently across siblings.
5. Re-run, then validate on held-out inputs.
6. If the question still won't behave, it is probably more than one judgment — split it and combine in code.

Related

• Choosing between Choice, Score, Noul — pick the type before you write the prompt
• Jev 1.13 jaggedness: known failure modes — the failure modes this guide works around
• Structured instructions, options, levels, criteria — the full structured-field contract
• Choice questions, Score questions, Noul (yes/no) questions — per-type contracts and worked examples
• Primitives: Choice, Score, Noul — batching and state paths
• State: what you send Jev — what to send and what to leave out
• Confidence vs probability — reading confidence without over-trusting it
• Testing and evaluating a Jev workflow — building the known-answer set
• HTTP API: POST /v1/systemone and GET /v1/models — exact field names and types

Sources

• raw/docs/primitives.md (https://docs.typesafe.ai/primitives)
• raw/docs/primitives__choice.md (https://docs.typesafe.ai/primitives/choice)
• raw/docs/primitives__score.md (https://docs.typesafe.ai/primitives/score)
• raw/docs/primitives__noul.md (https://docs.typesafe.ai/primitives/noul)
• raw/docs/primitives__advanced.md (https://docs.typesafe.ai/primitives/advanced)
• raw/docs/model-jaggedness__jev-1.13.md (https://docs.typesafe.ai/model-jaggedness/jev-1.13)
• raw/docs/api.md (https://docs.typesafe.ai/api)