Jev Case Study: RAG Chunk Reranking in Batch (Full Code)

TL;DR: This Jev case study adds a reranking step to a RAG pipeline: retrieve 8 chunks with embeddings, score each chunk’s relevance to the question with a Jev 1.13 scoring question (1-10 scale), and keep the top 3. Full batch script below.

Scenario

Our RAG pipeline answers questions over a few hundred product-doc pages. The embedding retrieval step is fast but crude: for a question like “how do I rotate an API key without downtime,” it happily returns the pricing page next to the actual rotation guide, because both mention “API key.”

The standard fix is a cross-encoder reranker. Ours was a second model we did not want to host, tune, and monitor. The lighter alternative we shipped instead: score each retrieved chunk with a Jev scoring question — “how relevant is this chunk to answering the question, 1 to 10?” — and keep the top 3.

The flow per user question:

1. Embeddings retrieve the top 8 candidate chunks (unchanged).
2. For each chunk, one Jev call returns {"answer": <1-10>, "scale": [1, 10], "confidence": <0-1>, "rationale": "..."}.
3. Sort by answer, keep the top 3, feed those to the answering step.

Jev is TypeSafe AI’s System One judgment model (September 2026), built for judgment, choice, and scoring questions; the scoring output is typed JSON, so the sort step never has to parse prose. This example runs through the OpenRouter OpenAI-compatible endpoint. The trade-off — 8 small calls per question — is a latency/cost question, covered under Notes.

Request

One call per chunk. For the chunk below:

{
  "model": "typesafe/jev-1.13",
  "messages": [
    {
      "role": "system",
      "content": "Scoring question. Score how relevant the chunk is to answering the user question, on a scale of 1 to 10. Reply with JSON only: {\"answer\": <1-10>, \"scale\": [1, 10], \"confidence\": <0-1>, \"rationale\": <one sentence>}"
    },
    {
      "role": "user",
      "content": "Question: How do I rotate an API key without downtime?\n\nChunk: To avoid downtime when rotating an API key, create the new key, deploy it alongside the old one, then revoke the old key after 24 hours."
    }
  ]
}

Response

Example fixture for the chunk above — illustrative output, not a live capture:

{
  "answer": 8,
  "scale": [1, 10],
  "confidence": 0.86,
  "rationale": "The chunk directly describes a zero-downtime key rotation procedure, though it does not cover revocation edge cases."
}

In the batch run, chunks that mention “API key” only in a pricing or marketing context score low (2-4 in our fixtures), which is exactly the separation the embedding step failed to make.

Reproduce

Save as rerank.py, set OPENROUTER_API_KEY, and run python3 rerank.py:

import json, os, urllib.request

URL = "https://openrouter.ai/api/v1/chat/completions"
MODEL = "typesafe/jev-1.13"  # confirm exact slug on the OpenRouter model page
QUESTION = "How do I rotate an API key without downtime?"

CHUNKS = [
    "Rotate keys from Settings > API keys. Old keys keep working for 24 hours, so you can deploy the new key first.",
    "Pricing for API usage is billed per million input tokens at the rate published on the pricing page.",
    "To avoid downtime when rotating an API key, create the new key, deploy it, then revoke the old key after 24 hours.",
    "Our support office hours are 9am-5pm PT, Monday through Friday, excluding public holidays.",
    "The API rate limit is 60 requests per minute on the standard plan and 600 on the enterprise plan.",
    "Two-factor authentication is required for all administrator accounts and cannot be disabled.",
    "Key rotation events are written to the audit log with the acting user and timestamp.",
    "Webhooks retry with exponential backoff for up to 24 hours before being marked failed.",
]

SYSTEM = ('Scoring question. Score how relevant the chunk is to answering the '
          'user question, on a scale of 1 to 10. Reply with JSON only: '
          '{"answer": <1-10>, "scale": [1, 10], "confidence": <0-1>, "rationale": <one sentence>}')

def score_chunk(chunk):
    body = {"model": MODEL, "messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "Question: " + QUESTION + "\n\nChunk: " + chunk}]}
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
        headers={"Authorization": "Bearer " + os.environ["OPENROUTER_API_KEY"],
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req) as res:
        payload = json.load(res)
    return json.loads(payload["choices"][0]["message"]["content"])

scored = [(score_chunk(c)["answer"], c) for c in CHUNKS]
top3 = [c for _, c in sorted(scored, key=lambda p: -p[0])[:3]]
print(json.dumps(top3, indent=2))

Key parameters

Notes

• Responses are example fixtures; verify field names against official docs before relying on them. Official API details live at typesafe.ai — see the System One announcement post.
• Confirm the exact model slug (typesafe/jev-1.13) on the OpenRouter model page before shipping.
• Latency: the script scores chunks sequentially, so 8 calls stack up. In production, run the per-chunk calls in parallel (they are independent) and budget for the slowest, not the sum.
• Treat confidence as a second signal: a chunk scored 8 with confidence 0.5 is weaker evidence than the same score at 0.9.
• Applies to Jev 1.13 (released September 2026) via the OpenRouter demonstration channel.