Jev AI Failure Modes: The 9 Jagged Edges TypeSafe Documents, and the Code That Fixes Each One

Updated Applies to Jev 1.14

TL;DR: TypeSafe publishes its own weakness list, at docs.typesafe.ai/model-jaggedness, and it names nine failure modes with a recommended fix for each: literal reading, math and numbers, date and time comparison, indirection, large state full of irrelevant detail, adversarial content, contradictory instructions and criteria, choice option order, and generation. Every one of them is a system-design problem, not a prompt-wording problem — the document’s own fix column says so, repeatedly pointing at “keep the arithmetic in code”, “filter first”, “reduce hops” and “use a generative model”. This page walks the nine in order, quotes what the documentation actually claims, and shows the shape of the code that absorbs each failure.

Most Jev guides on the internet describe what the model can do. This one is about what it cannot do, taken from the vendor’s own page rather than from guesswork, because the failure list is the part that decides whether your pipeline is safe to automate.

Every fact below was checked on 2026-11-13 against TypeSafe’s documentation: the model jaggedness page, the models page, the confidence page, the Choice primitive page, and two published cookbooks — date extraction and classifying RAG passages. Where a number appears below, it comes from one of those pages and is labelled with which one.

Why “jagged” is the right frame for a judgment model

The jaggedness page’s own framing is that Jev is fast, calibrated and good at common-sense judgment, but not perfect, and that it does the best on System One tasks. The sentence that matters most for architecture is the one after it: the model may struggle with tasks that require additional levels of indirection, it can be quite literal in its understanding, and it struggles with tasks that require numeric precision.

Read those three clauses as a capability boundary and the rest of the page becomes obvious. A judgment model is not a smaller LLM. It is a different instrument, and the nine failure modes are all places where someone tried to use a judgment instrument for a measurement. Counting is measurement. Date arithmetic is measurement. Interpolating a magnitude between two score levels is measurement. Generation is a different instrument entirely.

That is why the fixes are structural. When the documentation writes “keep the arithmetic in code” it is not asking you to phrase the prompt better; it is telling you that the question does not belong to this model at all.

The nine failure modes, as the page lists them

#Failure modeThe fix the documentation names
1Literal readingWrite the exact condition, criteria for each available option
2Math and numbersKeep the arithmetic in code
3Date and time comparisonExtract components; compare in code
4IndirectionReduce hops; point to the relevant state
5Large state full of irrelevant detailFilter first; send only what the question needs
6Adversarial contentWrite precise prompts, and test edge cases before deploying
7Contradictory instructions and criteriaAlign the criteria and instruction
8Choice option orderReorder the options and check the answer is consistent
9GenerationUse a generative model

The structure of that table is the useful part. Eight of the nine fixes happen outside the model — in your prompt, your state, or your code — and only one of them, option order, is a property you can only test rather than design away.

1. Literal reading: the model answers the question you wrote

The documentation’s phrasing is precise: Jev answers the question you wrote, not the one you meant, and scoping words, negations and implied conditions are read at face value. A question will be answered based on the words in the instruction, whereas a person might have read the intent behind the instructions.

The recommended fix doubles as a diagnostic. State the exact condition in instructions, be specific, put boundary cases in criteria, and — the sentence worth memorising — when you look at a wrong answer and find yourself explaining what you really meant, that explanation is the missing half of the instruction.

Two consequences follow.

First, boundary cases belong in criteria, not in prose of instructions. A Choice’s option descriptions are sent to the model alongside the option names (the Choice page says so explicitly: both the names and their descriptions are sent, so write descriptions that separate the options from each other). That means the separating detail — the one that decides “billing” versus “refund” — has a proper home, and it is not a preamble.

Second, when interpretation is genuinely unavoidable, split. The documentation’s own instruction is to split it into two literal questions and combine them in code. Two Nouls whose answers you and together is a literal query plus a code-level conjunction; one Noul containing “unless” is a bug waiting for the third edge case.

2. Math and numbers: keep the arithmetic in code

This is the failure mode with the most sub-clauses, because it covers three distinct mistakes.

Counting

The page states that Jev does not count reliably, that this covers characters in a word, occurrences of a term in a passage, and items in a long list, and that the model recognises the shape of an answer rather than tallying — with error growing with the size of the thing being counted.

It also asks a question worth asking about every counting request: why does the count need a model at all? If a regex or a parser can find the unit, the count belongs in code.

The documented pattern for the case that does need the model — count items matching a semantic criterion — is one question per item, summed in your code. This is the shape the documentation gives:

from typesafe_sdk import Noul, TypeSafeClient

client = TypeSafeClient(model="jev-latest")
YES = 0.5  # up to you on what you want the threshold to be, depends on your usecase.

items = ["typesafe", "apple", "california", "banana", "likes", "calibration", "orange", "vertex"]

result = client.system_one(
    {"items": items},
    {
        f"item_{i}": Noul(instructions=f"Is `items[{i}]` the name of a fruit?")
        for i in range(len(items))
    },
)

count = sum(result.nouls[f"item_{i}"].noul > YES for i in range(len(items)))

Note what is doing the work here: eight questions inside one request, one threshold in code, one sum in code. The model contributes eight semantic judgments; the arithmetic never enters the model. The comment on YES is the documentation’s, and it is honest — 0.5 is a starting point, not a recommended default.

Numeric representations

The second sub-clause is about how a value is written. The page says Jev performs better on semantic representations than numeric ones, gives the concrete example that questions about colours using hex values underperform those using English names, and adds that given RGB triples or hex values it cannot reliably judge whether two values are near each other. The same asymmetry is drawn for programming languages: high-level languages outperform questions about low-level assembly or binary-encoded instructions.

The fix is to convert in code and pass either the computed number or a named bucket. Keep the model for the part that is genuinely a judgment — whether a colour reads as a warning, say.

This is a useful general rule for state design: send {"severity_bucket": "high", "raw_score": 0.83} rather than only {"raw_score": 0.83}, because the bucket is the semantic representation and the raw number is the thing the model is worst at comparing.

Scores are not for magnitude

The third sub-clause forbids a specific habit. The documentation asks you not to use score outputs — expectations and probabilities — to compute the exact magnitude of a number between two levels of a criterion. Using the expectation to test a threshold is fine. Reconstructing the exact number by interpolating between the nearest two levels is not, because the page states that Jev’s score levels are weak in numerical calibration.

So expectation >= 0.6 is a legitimate use. 27.4 = 20 + (expectation − 0.5) × 2 × 10 dressed up as a measurement is not. If you need the actual number, compute it from data, not from a grade.

3. Date and time comparison: extract the parts, do the calendar in code

Jev reads dates as text, not as ordered quantities. The page states that asking which of two dates comes first, how far apart they are, or whether one falls inside a window is unreliable, and that it gets worse with mixed formats, relative references and domain boundaries such as quarters, settlement windows and accrual periods.

The recommended split is clean, and it is the best example on the whole page of how to divide labour between a judgment model and ordinary code: extraction is a judgment, so give it to the model; arithmetic is not, so keep it in code.

Why extraction can be a judgment at all is the elegant part. Every part of a date is a small closed set — twelve months, thirty-one possible days, a bounded range of years — so extraction becomes a Choice over enumerated options rather than free-form parsing, and an explicit “not stated” option gives a missing part somewhere to go so it is reported rather than guessed.

The date-extraction cookbook is the worked version. Two details from it are worth carrying into any implementation:

The cookbook also names the ambiguity you must resolve yourself: “next Thursday” can mean two different days, so code decides which — a bare weekday means the next one, and the resolution convention is written in code, not inferred by the model. That is the whole pattern in one sentence. The model reads what the text says; your code owns the calendar.

4. Indirection: fewer hops, named parts

Instructions carrying double negatives or complex indirection are answered less reliably, and the page says a question about a property of a property, or one requiring multiple hops of reasoning, costs accuracy. Its fix is to write instructions as directly as possible and, when possible, to identify the relevant parts of state by name.

That last clause is a concrete design instruction. If the relevant clause lives at state.contract.clauses[7].text, say so, or restructure so the question points at a named field. “Was the notice period honoured?” that must traverse three levels is a System Two task wearing a Noul’s clothes — and the jaggedness page’s closing reminder names exactly that: System Two tasks, more layers of indirection, are on the avoid list.

5. Large state full of irrelevant detail: filter first, and mind context rot

Accuracy falls as the state grows with content unrelated to the decision. The page’s own explanation is that unrelated detail acts as a distractor, and that a large state makes it harder to tell which part of the input produced a wrong answer — which is a debugging cost as much as an accuracy cost.

The note under that section uses the term directly: Jev suffers from context rot, so unrelated material in the state costs you accuracy. The models page repeats the point from the other direction, linking to the jaggedness page from its context-length section as the account of how accuracy shifts as the state grows.

Two fixes are named. Retrieve and filter in code first, sending only the fields the question needs. And when you genuinely cannot filter in state, use a Noul to filter for relevance — the classifying-RAG-passages cookbook is the worked example.

That cookbook is also the most concrete evidence on the whole page for why relevance filtering is not optional. In its first query, twelve passages were retrieved and their similarity scores fell between 0.584 and 0.455 — a spread the cookbook itself describes as too narrow to separate the passage that corrects the query from the one trying to hijack the answer. The forum post carrying an injected instruction ranked first at 0.584; the passage that refutes the query’s premise ranked seventh at 0.509. Retrieval order did not solve it. Four judgment questions per passage did.

6. Adversarial content: state is data, and it is not treated as hostile

The documentation states that state is data and that Jev does not treat it as hostile by default. Content written to adversarially steer the model — an injected instruction, a deliberately misleading framing, or text that argues for its own classification — can move the answer. The page adds that the team expects to improve on this in the future, which is the honest version of “this is not a solved problem”.

The stated mitigation is two-part and both parts are your responsibility: be explicit in the criteria, and test your integration thoroughly before deploying it to many users. No threshold and no header makes injection go away.

An important nuance from the same cookbook: the natural defence — a probability threshold on an injection question — is a policy number, not a validated constant. The cookbook’s own THRESHOLDS block uses injection_max: 0.70 and warns, in a note of its own, that the four numbers were picked for that corpus and should be treated as a starting point rather than defaults. It also explains the ordering of its routing tests, which is the design lesson: injection is checked first, because it is a security decision rather than an evidence one; the contradiction test comes before the evidence test because a passage that denies the query’s premise can state something usable too, and checked the other way round it would land in the accepted block.

That ordering is the difference between a filter and a wish. Order tests by consequence, not by convenience.

7. Contradictory instructions and criteria: the criteria are an extension of the instruction

When instructions and criteria ask for different things, Jev might get confused. The page’s example is specific and worth quoting in substance: a Noul where true maps to no and false maps to yes will perform worse. Its guidance is to aim for instructions that are easy for the average person to read and understand, and to treat the criteria as an extension of the instruction, aligned through clear and precise language.

The practical audit is mechanical: read instructions, then read every criteria key and description, and check that a human reading only those two fields would give the same answer you expect. Inverted polarity — the true-means-no case — is the one to grep for, because it usually arrives as a well-intentioned “just reverse it in code” shortcut.

8. Choice option order: leaning toward the first option, so test it

In some cases the order of a Choice’s options can affect the answer, and the page states that Jev leans toward the option that comes first. The recommended check: reorder the options and confirm the answer stays consistent.

This is the only failure mode on the list whose fix is purely a test, which tells you something about its status. There is no design rule that removes it; there is a verification step that detects it. If a choice carries real consequences — a routing decision, a risk tier — run your labelled set twice, once in the shipped order and once shuffled, and compare agreement. If the answer moves with the order, the option set is probably not separating the options well enough, which takes you back to failure mode 1 and the instruction to write criteria that do the separating.

9. Generation: use a generative model

The page is unambiguous. Jev is not trained to generate text. You can force it by chaining choices, but that will not work well and will be very slow. For data extraction, extract candidate options with regex or a generative model and let Jev pick the correct extraction. When the answer space is bounded, turn extraction into a Choice over the options rather than asking for the value itself.

If you genuinely need to generate text, the documentation’s own conclusion is that there are other models for that. That is not a limitation to work around; it is the division of labour that makes the cost model work in the first place — you pay per input token and output tokens are free (models page, checked 2026-11-13), which is a pricing structure that only makes sense for a model that answers rather than writes.

The four reminders at the end of the page

The jaggedness page closes with a short “avoid the following” list, and because it reads like a summary of everything above, it is the best single checklist on the site:

Every one of these is detectable in code review. That is the point: the failure list is a review checklist, not a disclaimer.

Turning this into a pre-launch audit

Because all nine fixes are structural, they can be checked before a single request is billed. A workable order:

  1. List every question and label its type. For each, write down what kind of answer it is: a judgment, a measurement, or a generation. Anything labelled measurement or generation is a finding, and moves to code or to another model.
  2. Grep the state construction for raw numbers. Hex colours, RGB triples, unparsed timestamps, and any value the question compares against a threshold. Add a named bucket or move the comparison into code.
  3. Check every date question. If the answer depends on ordering, duration or a window, the question should be extracting parts, not comparing them.
  4. Shuffle the choice options and diff the answers. Ship whichever order you like, but know whether the answer moves.
  5. Read instructions against criteria line by line. Look for inverse polarity and for conditions stated in prose that should be criteria.
  6. Measure state size against question count. Anything in the state that no question references is a distractor by definition — the jaggedness page’s context-rot note says it costs accuracy, and the filter is cheap.
  7. Run the adversarial set. Texts that argue for their own classification, a misleading framing, an injected instruction. Check the resulting route, not just the raw probability.

What this changes about the architecture

The consistent implication across all nine modes is that Jev should sit at the narrowest point of your pipeline — a decision that has already been narrowed by code. Retrieval decides what enters the state. Code decides what the numbers mean. The model decides the one thing code cannot: whether this text means what the criteria say it means.

Read that way, the jaggedness page is not a list of bugs. It is a description of where the boundary is, published by the vendor, with the fixes attached. Most of the design work for a reliable Jev integration is the work of respecting it.

One caveat on the page itself: it opens by saying many of the failure modes will be fixed in later versions. Treat the list as dated. The version this page was written against is 1.14, the newest release as of the models page checked on 2026-11-13, and the changelog is the place to check whether any of these clauses changed before you cite them in your own architecture review.

Frequently asked questions

+ Does TypeSafe publish a list of Jev's weaknesses?

Yes. It is a documentation page titled Model jaggedness, at docs.typesafe.ai/model-jaggedness, and its own summary line reads that Jev is not perfect and that the page lists jagged edges the company is aware of. The page opens by saying many of them will be fixed in later versions, so the list is a snapshot rather than a fixed spec — re-read it when you upgrade.

+ What are the nine Jev failure modes?

The page's table lists them in this order: literal reading, math and numbers, date and time comparison, indirection, large state full of irrelevant detail, adversarial content, contradictory instructions and criteria, choice option order, and generation. For each one the page names a fix, which is the column worth reading first.

+ Can Jev count items or do arithmetic?

No, and the documentation is blunt about it. It states that Jev is not a calculator, that it does not count reliably — characters in a word, occurrences of a term, items in a long list — and that the model recognises the shape of an answer rather than tallying, with the error growing as the thing being counted grows. The recommended fix is to count in code and, when you need per-item judgment, ask one question per item and add the answers up yourself.

+ Why does adding more context to the state make Jev less accurate?

The jaggedness page says accuracy falls as the state grows with content unrelated to the decision, because unrelated detail acts as a distractor and a large state makes it harder to tell which part of the input produced a wrong answer. Its own note calls this context rot. The fix is to retrieve and filter in code first, or to use a Noul to filter for relevance when you cannot filter in state, and the models page repeats the point by linking to it from the context-length section.

+ Is Jev vulnerable to prompt injection in the state?

The documentation says state is treated as data and is not treated as hostile by default, and that content written to adversarially steer the model — an injected instruction, a misleading framing, or text arguing for its own classification — can move the answer. It also says the team expects to improve on this. The stated mitigation is precise criteria plus thorough testing before deploying to many users, not a guarantee that the model will ignore injected text.

+ Does the order of Choice options change Jev's answer?

Sometimes. The jaggedness page states that in some cases the order of a Choice's options can affect the answer and that Jev leans toward the option that comes first. The recommended check is to reorder the options and confirm the answer stays consistent — which is a test, not a guarantee, so treat option order as something to verify rather than assume.

+ Should I ask Jev to generate text?

No. The documentation states plainly that Jev is not trained to generate text, that forcing it by chaining choices will not work well and will be very slow, and that for data extraction you should extract candidate options with regex or a generative model and let Jev pick the correct one. When the answer space is bounded, the recommended shape is a Choice over the options rather than a request for the value itself.

Keep reading