Jev AI Infinite-Cardinality Choice: Picking One Item Out of 5,000 in Three Rounds

Updated Applies to Jev 1.14

A Choice question picks one option from a set you hand it. That works beautifully for a handful of teams, a dozen categories, or thirty product lines. It does not work for a catalog of 5,000 films, 40,000 SKUs, or a support knowledge base that grows every week — not because Jev cannot judge 5,000 things, but because one question can only carry so many options and one request can only hold so many tokens.

TypeSafe documents the way out of that corner in a cookbook called Infinite cardinality choice, and it is unusually well measured: 100 real requests, four strategies compared on the same data, and a per-band accuracy table for setting the threshold. This page walks the three rounds in the order the code runs them, keeps every number attached to the run that produced it, and ends with the arithmetic you need before you copy the shape into your own catalog.

Provenance: every figure, threshold, constant and code shape below comes from TypeSafe’s docs.typesafe.ai/cookbooks/infinite_cardinality_choice, docs.typesafe.ai/primitives/choice and docs.typesafe.ai/models, read on 2026-11-18. The recorded probabilities are the ones the published run printed. Nothing on this page is an invented illustration.

The two ceilings you hit first

Two documented limits make a 5,000-item Choice impossible, and they are not the same limit.

The option count. A Choice question accepts up to 255 options (docs.typesafe.ai/primitives/choice). For short options the docs actively advise passing the full list rather than a shortlist, because each option costs only a few tokens. The cookbook shows the other side of that: when each option is a long plot summary, the practical ceiling collapses to roughly forty options.

The context window. A Jev request must fit 64k tokens per request and 32k per question, as listed on the models page for both jev-1.14.0 and jev-1.13.0. A single item described in 600 words of plot is about 700 tokens, so forty of them already fill roughly 28k — under the per-request limit but well past the point where adding a forty-first option is comfortable.

So the real question is not “how do I make Jev rank 5,000 options”. It is how do I get from 5,000 down to 20 cheaply, and only then pay for a ranking. The cookbook answers with three rounds that read more about fewer items each time.

The data the cookbook measured on

The numbers below are only meaningful against the dataset that produced them, so here it is, as documented.

The requests read like a person describing a film they half-remember. Request 388, for instance, describes a mother and two kids tormented by ghosts that turn out to be time travellers, and the film is The Diabolical (2015).

Round 1 — one Noul per film, over the short plot

Round one is deliberately not a Choice. It is 5,000 Noul questions, one per film, each phrased as a proposition with that film’s short plot inside it:

The film this person is trying to remember is {title} ({year}).
Its plot: {short plot}

The wording matters: the question compares the request against the plot text instead of asking Jev to recall a film from its title. Title recall is exactly the thing a tip-of-the-tongue request defeats.

Because a call holds at most 64k tokens, the catalog is split into chunks — CHUNK = 300 films per call, since ~150 tokens each keeps a call well under the limit — and all chunks go out at once rather than in sequence. No chunk depends on another, so a ThreadPoolExecutor sized to the number of chunks sends them together. The published run reports:

5,000 answers from 17 calls sent together; the slowest took 0.4s

Seventeen calls, because 5,000 / 300 rounds up to 17. The wait stays roughly flat as the catalog grows, as long as your rate limit allows that many calls at a time — which is the real constraint, not the latency. Over the 100 requests, the filter’s own top eighty hold the right film for 93 of them.

The filter is good but not precise, and the worked example shows why: for request 388 the filter’s top five were Ghostbusters: Afterlife at 0.64, The Curse of La Llorona at 0.37, Thirteen Ghosts at 0.33, Mama at 0.31 and Wicked Little Things at 0.31 — and The Diabolical sat at position six. It matched on the surface facts (a mother, two kids, a haunted house, money trouble). The detail that actually identifies the film — the ghosts being time travellers — appears in the plot after the first hundred words, so the filter never saw it.

Round 2 — re-read the top eighty with long plots

So the second round asks about the filter’s top FIRST_CUT = 80 films again, this time with six hundred words of plot each. Longer questions mean fewer films per chunk: LONG_CHUNK = 40, roughly 700 tokens each, about 28k per call. The chunks still go out together.

For request 388 this is what moves The Diabolical to the front:

the long plots' top five:
  0.60  The Diabolical (2015)  <- the film
  0.40  Ghostbusters: Afterlife (2021)
  0.30  The Others (2001)
  0.26  Thirteen Ghosts (2001)
  0.17  Knowing (2009)

Across the hundred requests, the top twenty after re-reading hold the right film for 88, against 82 for the filter’s top twenty alone. Round two buys six points of recall for roughly a seventieth of round one’s tokens — and, as the next section shows, it also changes what the final Choice question is able to see.

Round 3 — one Choice over twenty

Round three is the only Choice question in the pipeline, and it is small: CANDIDATES = 20 films, each option labelled with the title and the long plot, all weighed against each other in a single question.

Which film is this person trying to remember?
  the_diabolical: The Diabolical (2015): <600-word plot>
  ghostbusters_afterlife: Ghostbusters: Afterlife (2021): <600-word plot>
  ...

The published answer for request 388 puts 1.00 on The Diabolical and 0.00 on the other nineteen. A Choice is decisive in a way a Noul cannot be, because it normalises across the options.

That difference is the reason the pipeline bothers with all three rounds instead of stopping at a shortlist. Each Noul judges its film alone, so two films that both fit can both score 0.4 or higher — in the worked example 0.60 and 0.40, which add up to more than 1 because they answer different questions. A Choice over the same twenty answers one question: how far ahead of the rest is this one. Its top probability is therefore directly readable as “the chance the top item is right”, which is what a threshold needs.

The comparison that decides whether any of this is worth it

The cookbook runs the same Choice question three ways on all 100 requests: with a shortlist from embedding search, with the filter’s top twenty and no re-read, and with all three rounds.

embeddingsno re-readall 3 steps
reaches the Choice35%82%88%
film in the top 133%73%78%
film in the top 334%76%82%

The embeddings column is the standard first try: embed each film’s short plot and the request with OpenAI’s text-embedding-3-small, and send the twenty closest films to the same Choice question. It reaches the Choice with the right film 35% of the time, and the film comes out first 33% of the time.

Read the table as a chain of ceilings. The Choice question can only rank the films it is given, so shortlist recall bounds everything downstream: 33% top-one from embeddings is essentially the 35% recall, rounded. Jev reading short plots against the request raises shortlist recall to 82% and top-one to 73%. Re-reading with long plots adds the last six points of recall and five of top-one — the smallest gain in the table for the smallest token spend.

Where the threshold goes: on the Choice, not the Noul

A search that returns ranked films still has to decide, per request, whether to show one film or a list. The cookbook’s answer is to threshold the Choice question’s top probability, and the band table is the evidence:

top answerlong-plot NoulsChoice question
under 0.530/49 right9/30 right
0.5 to 0.833/40 right21/22 right
0.8 to 0.9511/11 right18/18 right
0.95 and up0/0 right30/30 right

Two things fall out of that table.

The Choice sorts requests cleanly. At 0.8 or more it was right in 48 of 48 cases, so the page shows one film. Below 0.8 the top three held the right film for 34 of the remaining 52, so the page shows a short list. Under 0.5 the Choice is right 9 times in 30 — noise, and the code should never present it as an answer.

The Noul probabilities cannot make that split, even though they are calibrated. None of the re-read’s answers reaches 0.95, only 11 reach 0.8, and answers under 0.5 are still right 30 times in 49. There is no band where a Noul threshold separates reliable from unreliable, which is why “threshold the Choice top probability” is the documented rule here and not “threshold whatever number came back”.

The four constants, and how to size them

ConstantPublished valueWhy that value
CHUNK (round 1 items per call)300~150 tokens per film keeps a call well under 64k
LONG_CHUNK (round 2 items per call)40~700 tokens per film, ~28k per call
FIRST_CUT (items re-read)80holds the right film 93/100 before re-reading
CANDIDATES (items in the Choice)20holds the right film 88/100 after re-reading
WORKERS (searches side by side)4each search still sends all its own chunks at once

Only the first two are hard ceilings from documented context limits; the last three are tuning, and the cookbook’s own numbers tell you which direction each one is worth pushing. Raising FIRST_CUT costs tokens linearly in round two and moves recall slowly. Raising CANDIDATES is nearly free in tokens but pushes toward the ~40-option comfort limit for long options.

What one search costs in tokens

The docs price Jev at $42 per Btok — that is, $0.042 per Mtok — charged on input tokens only, output free. Applying that rate to the cookbook’s own per-item token estimates gives a rough bill for one search over 5,000 films:

This is arithmetic on the documented token estimates and list price, not a measured bill — the cookbook reports accuracy, not cost. The shape holds regardless of the exact number: the filter round is ~91% of the spend, so if you need to cut cost, shrink the catalog or shorten the short descriptions rather than dropping round three.

Porting the shape to your own catalog

The cookbook is explicit about the seam: “To search your own catalog, give each item a short and a long description and rewrite FILMS, proposition() and decision_question(). Nothing else mentions films.”

That is the whole port. What you supply is two descriptions per item — a short one that names it and carries enough surface detail for a coarse pass, and a long one that carries the distinguishing detail. The short/long split is not decorative; the worked example shows the identifying fact living past the hundred-word mark, which is precisely the reason round two exists.

Two boundary conditions worth stating plainly:

One more constraint that bites at scale: rate limits, not latency. Requests over the token-per-second or requests-per-second limits return 429 Too Many Requests, and the SDKs retry with backoff, honouring retry-after. Round one is 17 parallel calls per search; at four searches side by side that is 68 concurrent calls, so a catalog-wide sweep is a rate-limit question before it is a model question.

FAQ

How many options can one Jev Choice question hold? 255, per the Choice primitive page — and each option costs a few tokens. With long options the effective ceiling in the cookbook’s own pipeline is about forty.

Why not just use embedding search to build the shortlist? Because the shortlist bounds everything downstream. The same Choice question, given embedding search’s twenty films, put the right film first 33% of the time; the three-round search put it first 78%.

Where should the “show one answer” threshold go? On the Choice question’s top probability, at 0.8 — right 48 times out of 48 in the published run. Below that, show the top three, which held the answer for 34 of the remaining 52.

Does any of this need Jev 1.14? No. Noul and Choice exist on both 1.13 and 1.14; the search uses no Box question, no images and no long-context header.

Sources

Frequently asked questions

+ How many options can one Jev Choice question hold?

The limit is 255 options per Choice question, documented on the Choice primitive page, and every option costs a few tokens. That ceiling is about short options. In the infinite-cardinality cookbook, each option is a long plot summary and the effective limit drops to about forty. Anything past that has to be reached in rounds, not in one question.

+ Why not just use embedding search to build the shortlist?

Because the shortlist is the ceiling on everything after it. In the cookbook the same Choice question ranked twenty films for 100 requests: embedding search put the right film in its twenty for 35% of them and the film came out first for 33%, while the three-round Jev search reached 88% into the Choice and put the film first for 78%.

+ What probability threshold decides between showing one answer and showing a list?

Use the Choice question's top probability, not the Noul probabilities. At 0.8 or more the top film was right in all 48 of the 48 requests that reached it, so the page shows one film; below 0.8 the top three held the right film for 34 of the remaining 52. The re-read's Noul probabilities could not make that split: none reached 0.95, only 11 reached 0.8, and answers under 0.5 were still right 30 times in 49.

+ Does a three-round search work on Jev 1.13?

Yes. Noul and Choice are question types on both 1.13 and 1.14, and the cookbook's search uses only those two — no Box, no images, no long context header. Nothing in the three rounds needs a 1.14-only feature.

Keep reading