Jev AI Infinite-Cardinality Choice: Picking One Item Out of 5,000 in Three Rounds
A Choice question picks one option from a set you hand it. That works beautifully for a handful of teams, a dozen categories, or thirty product lines. It does not work for a catalog of 5,000 films, 40,000 SKUs, or a support knowledge base that grows every week — not because Jev cannot judge 5,000 things, but because one question can only carry so many options and one request can only hold so many tokens.
TypeSafe documents the way out of that corner in a cookbook called Infinite cardinality choice, and it is unusually well measured: 100 real requests, four strategies compared on the same data, and a per-band accuracy table for setting the threshold. This page walks the three rounds in the order the code runs them, keeps every number attached to the run that produced it, and ends with the arithmetic you need before you copy the shape into your own catalog.
Provenance: every figure, threshold, constant and code shape below comes from TypeSafe’s
docs.typesafe.ai/cookbooks/infinite_cardinality_choice,docs.typesafe.ai/primitives/choiceanddocs.typesafe.ai/models, read on 2026-11-18. The recorded probabilities are the ones the published run printed. Nothing on this page is an invented illustration.
The two ceilings you hit first
Two documented limits make a 5,000-item Choice impossible, and they are not the same limit.
The option count. A Choice question accepts up to 255 options (docs.typesafe.ai/primitives/choice). For short options the docs actively advise passing the full list rather than a shortlist, because each option costs only a few tokens. The cookbook shows the other side of that: when each option is a long plot summary, the practical ceiling collapses to roughly forty options.
The context window. A Jev request must fit 64k tokens per request and 32k per question, as listed on the models page for both jev-1.14.0 and jev-1.13.0. A single item described in 600 words of plot is about 700 tokens, so forty of them already fill roughly 28k — under the per-request limit but well past the point where adding a forty-first option is comfortable.
So the real question is not “how do I make Jev rank 5,000 options”. It is how do I get from 5,000 down to 20 cheaply, and only then pay for a ranking. The cookbook answers with three rounds that read more about fewer items each time.
The data the cookbook measured on
The numbers below are only meaningful against the dataset that produced them, so here it is, as documented.
films.jsonlholds 5,000 films, each with a title, a year, and two versions of the plot section of its Wikipedia article, fetched 2026-11-07. The short one,plot, is the first hundred or so words. The long one,long_plot, is the first six hundred or so. The catalog holds the film behind every request; the films a search has to rule out are those whose Wikipedia articles exist in the most languages.requests.jsonlholds 100 requests from Microsoft’s tip-of-the-tongue dataset, collected from the forumiRememberThisMovie.com, each with the film that answered it. They are a random sample of the 709 whose film has a plot section on Wikipedia.- The model is
jev-1.14.0. The notebook “runs as published without an API key”, because the published answers are cached — so every probability quoted here is a recorded answer, not a re-run on a different day.
The requests read like a person describing a film they half-remember. Request 388, for instance, describes a mother and two kids tormented by ghosts that turn out to be time travellers, and the film is The Diabolical (2015).
Round 1 — one Noul per film, over the short plot
Round one is deliberately not a Choice. It is 5,000 Noul questions, one per film, each phrased as a proposition with that film’s short plot inside it:
The film this person is trying to remember is {title} ({year}).
Its plot: {short plot}
The wording matters: the question compares the request against the plot text instead of asking Jev to recall a film from its title. Title recall is exactly the thing a tip-of-the-tongue request defeats.
Because a call holds at most 64k tokens, the catalog is split into chunks — CHUNK = 300 films per call, since ~150 tokens each keeps a call well under the limit — and all chunks go out at once rather than in sequence. No chunk depends on another, so a ThreadPoolExecutor sized to the number of chunks sends them together. The published run reports:
5,000 answers from 17 calls sent together; the slowest took 0.4s
Seventeen calls, because 5,000 / 300 rounds up to 17. The wait stays roughly flat as the catalog grows, as long as your rate limit allows that many calls at a time — which is the real constraint, not the latency. Over the 100 requests, the filter’s own top eighty hold the right film for 93 of them.
The filter is good but not precise, and the worked example shows why: for request 388 the filter’s top five were Ghostbusters: Afterlife at 0.64, The Curse of La Llorona at 0.37, Thirteen Ghosts at 0.33, Mama at 0.31 and Wicked Little Things at 0.31 — and The Diabolical sat at position six. It matched on the surface facts (a mother, two kids, a haunted house, money trouble). The detail that actually identifies the film — the ghosts being time travellers — appears in the plot after the first hundred words, so the filter never saw it.
Round 2 — re-read the top eighty with long plots
So the second round asks about the filter’s top FIRST_CUT = 80 films again, this time with six hundred words of plot each. Longer questions mean fewer films per chunk: LONG_CHUNK = 40, roughly 700 tokens each, about 28k per call. The chunks still go out together.
For request 388 this is what moves The Diabolical to the front:
the long plots' top five:
0.60 The Diabolical (2015) <- the film
0.40 Ghostbusters: Afterlife (2021)
0.30 The Others (2001)
0.26 Thirteen Ghosts (2001)
0.17 Knowing (2009)
Across the hundred requests, the top twenty after re-reading hold the right film for 88, against 82 for the filter’s top twenty alone. Round two buys six points of recall for roughly a seventieth of round one’s tokens — and, as the next section shows, it also changes what the final Choice question is able to see.
Round 3 — one Choice over twenty
Round three is the only Choice question in the pipeline, and it is small: CANDIDATES = 20 films, each option labelled with the title and the long plot, all weighed against each other in a single question.
Which film is this person trying to remember?
the_diabolical: The Diabolical (2015): <600-word plot>
ghostbusters_afterlife: Ghostbusters: Afterlife (2021): <600-word plot>
...
The published answer for request 388 puts 1.00 on The Diabolical and 0.00 on the other nineteen. A Choice is decisive in a way a Noul cannot be, because it normalises across the options.
That difference is the reason the pipeline bothers with all three rounds instead of stopping at a shortlist. Each Noul judges its film alone, so two films that both fit can both score 0.4 or higher — in the worked example 0.60 and 0.40, which add up to more than 1 because they answer different questions. A Choice over the same twenty answers one question: how far ahead of the rest is this one. Its top probability is therefore directly readable as “the chance the top item is right”, which is what a threshold needs.
The comparison that decides whether any of this is worth it
The cookbook runs the same Choice question three ways on all 100 requests: with a shortlist from embedding search, with the filter’s top twenty and no re-read, and with all three rounds.
| embeddings | no re-read | all 3 steps | |
|---|---|---|---|
| reaches the Choice | 35% | 82% | 88% |
| film in the top 1 | 33% | 73% | 78% |
| film in the top 3 | 34% | 76% | 82% |
The embeddings column is the standard first try: embed each film’s short plot and the request with OpenAI’s text-embedding-3-small, and send the twenty closest films to the same Choice question. It reaches the Choice with the right film 35% of the time, and the film comes out first 33% of the time.
Read the table as a chain of ceilings. The Choice question can only rank the films it is given, so shortlist recall bounds everything downstream: 33% top-one from embeddings is essentially the 35% recall, rounded. Jev reading short plots against the request raises shortlist recall to 82% and top-one to 73%. Re-reading with long plots adds the last six points of recall and five of top-one — the smallest gain in the table for the smallest token spend.
Where the threshold goes: on the Choice, not the Noul
A search that returns ranked films still has to decide, per request, whether to show one film or a list. The cookbook’s answer is to threshold the Choice question’s top probability, and the band table is the evidence:
| top answer | long-plot Nouls | Choice question |
|---|---|---|
| under 0.5 | 30/49 right | 9/30 right |
| 0.5 to 0.8 | 33/40 right | 21/22 right |
| 0.8 to 0.95 | 11/11 right | 18/18 right |
| 0.95 and up | 0/0 right | 30/30 right |
Two things fall out of that table.
The Choice sorts requests cleanly. At 0.8 or more it was right in 48 of 48 cases, so the page shows one film. Below 0.8 the top three held the right film for 34 of the remaining 52, so the page shows a short list. Under 0.5 the Choice is right 9 times in 30 — noise, and the code should never present it as an answer.
The Noul probabilities cannot make that split, even though they are calibrated. None of the re-read’s answers reaches 0.95, only 11 reach 0.8, and answers under 0.5 are still right 30 times in 49. There is no band where a Noul threshold separates reliable from unreliable, which is why “threshold the Choice top probability” is the documented rule here and not “threshold whatever number came back”.
The four constants, and how to size them
| Constant | Published value | Why that value |
|---|---|---|
CHUNK (round 1 items per call) | 300 | ~150 tokens per film keeps a call well under 64k |
LONG_CHUNK (round 2 items per call) | 40 | ~700 tokens per film, ~28k per call |
FIRST_CUT (items re-read) | 80 | holds the right film 93/100 before re-reading |
CANDIDATES (items in the Choice) | 20 | holds the right film 88/100 after re-reading |
WORKERS (searches side by side) | 4 | each search still sends all its own chunks at once |
Only the first two are hard ceilings from documented context limits; the last three are tuning, and the cookbook’s own numbers tell you which direction each one is worth pushing. Raising FIRST_CUT costs tokens linearly in round two and moves recall slowly. Raising CANDIDATES is nearly free in tokens but pushes toward the ~40-option comfort limit for long options.
What one search costs in tokens
The docs price Jev at $42 per Btok — that is, $0.042 per Mtok — charged on input tokens only, output free. Applying that rate to the cookbook’s own per-item token estimates gives a rough bill for one search over 5,000 films:
- Round 1: 5,000 films × ~150 tokens ≈ 751k tokens (including the request text repeated across 17 calls) ≈ $0.032
- Round 2: 80 films × ~700 tokens ≈ 56k tokens ≈ $0.0024
- Round 3: 20 films × ~700 tokens ≈ 14k tokens ≈ $0.0006
- Total ≈ 822k tokens ≈ $0.035 per search, so roughly $3.45 per 100 searches
This is arithmetic on the documented token estimates and list price, not a measured bill — the cookbook reports accuracy, not cost. The shape holds regardless of the exact number: the filter round is ~91% of the spend, so if you need to cut cost, shrink the catalog or shorten the short descriptions rather than dropping round three.
Porting the shape to your own catalog
The cookbook is explicit about the seam: “To search your own catalog, give each item a short and a long description and rewrite FILMS, proposition() and decision_question(). Nothing else mentions films.”
That is the whole port. What you supply is two descriptions per item — a short one that names it and carries enough surface detail for a coarse pass, and a long one that carries the distinguishing detail. The short/long split is not decorative; the worked example shows the identifying fact living past the hundred-word mark, which is precisely the reason round two exists.
Two boundary conditions worth stating plainly:
- If an answer may not be in your catalog at all, a three-round search will still hand you a ranked list. The cookbook’s catalog guarantees containment — every request’s film is in the 5,000 — which is why it adds no “none of these” option. When containment is not guaranteed, the documented remedy is a separate Noul asking whether anything fits, read before the Choice; a catch-all
otheroption instead competes with real options for probability. See our request-patterns guide for the recorded numbers on that failure. - If your catalog is small, skip the rounds. Under 255 short options, a single Choice question is the documented shape and the docs advise passing the whole list. Rounds buy nothing when the option limit was never the problem.
One more constraint that bites at scale: rate limits, not latency. Requests over the token-per-second or requests-per-second limits return 429 Too Many Requests, and the SDKs retry with backoff, honouring retry-after. Round one is 17 parallel calls per search; at four searches side by side that is 68 concurrent calls, so a catalog-wide sweep is a rate-limit question before it is a model question.
FAQ
How many options can one Jev Choice question hold? 255, per the Choice primitive page — and each option costs a few tokens. With long options the effective ceiling in the cookbook’s own pipeline is about forty.
Why not just use embedding search to build the shortlist? Because the shortlist bounds everything downstream. The same Choice question, given embedding search’s twenty films, put the right film first 33% of the time; the three-round search put it first 78%.
Where should the “show one answer” threshold go? On the Choice question’s top probability, at 0.8 — right 48 times out of 48 in the published run. Below that, show the top three, which held the answer for 34 of the remaining 52.
Does any of this need Jev 1.14? No. Noul and Choice exist on both 1.13 and 1.14; the search uses no Box question, no images and no long-context header.
Sources
docs.typesafe.ai/cookbooks/infinite_cardinality_choice— dataset description, three rounds, all comparison and band tables, dataset fetched 2026-11-07, read 2026-11-18.docs.typesafe.ai/primitives/choice— the 255-option limit and the advice to pass full lists for short options.docs.typesafe.ai/models—$42/Btok($0.042/Mtok), output tokens free, 64k per-request and 32k per-question context onjev-1.14.0andjev-1.13.0, 429 behaviour.docs.typesafe.ai/patterns/companion-noul— the documented remedy when a Choice’s option set may not contain the answer.