Jev reranking for RAG: keep the passages that answer the question

The problem

Vector search returns things that are semantically near but not useful. Stuffing all of them into context costs tokens and dilutes the answer, while a cross-encoder reranker is another model to host and tune.

How Jev handles it

Score each candidate passage against a relevance rubric. The calls are cheap and fast enough to run across your whole candidate set, and you get a distribution to threshold rather than an opaque similarity number.

RELEVANCE = Score(
    instructions="How useful is this passage for answering the question?",
    criteria=["Irrelevant", "Tangential", "Useful", "Directly answers"],
)

def rank(passage):
    response = client.system_one(
        model="jev-latest",
        state=f"QUESTION: {question}\n\nPASSAGE: {passage.text}",
        questions={"relevance": RELEVANCE},
    )
    return response.answers["relevance"]

scored = [(p, rank(p)) for p in candidates]
# score is a 0..3 weighted mean: 2 = "Useful", 3 = "Directly answers"
keep = [p for p, s in scored if round(s.score) >= 2]

Wording the relevance rubric for Jev reranking

Four ordered levels give Jev reranking two useful cut points: drop everything below Useful, and keep only Directly answers when the context budget is tight. The boundary that matters most is Tangential against Useful — same topic, but does it help? — so spend your wording there. The measured passage is the textbook near-miss: it explains where API keys are created when the question asks how to rotate one without downtime, and 0.86 of the probability went to Tangential.

Sort the survivors on the raw score rather than the rounded band. A 2.9 and a 2.1 both round to Useful and are not equally good, and Jev's weighted mean orders them for you. When your corpus mixes product versions or regions, a Noul asking whether the passage applies to the version named in the question catches a mismatch that similarity search cannot see.

What to put in the state

Jev reranking judges one passage against one question, so keep the fixture's two labelled parts, QUESTION and PASSAGE, and pass the user's question verbatim rather than the rewritten search query — the rubric asks about answering the question, so the question is what it needs. Add the source document's title and section heading where you have them; a breadcrumb like "Settings > API" tells Jev where a chunk came from.

Do not include neighbouring chunks: they turn one judgement into several. Chunk size is the real lever, because an oversized chunk buries the one relevant sentence among unrelated text, and every token of it is billed once per passage per query.

What one decision costs

Jev reranking for RAG multiplies calls by the size of the candidate set, so price a query, not a call. First line: the measured passage. Second: one query with 20 candidates of that size, one call each. Third: 100,000 such queries a month.

# the measured passage, one Score
364 tokens × $0.042 / 1M = $0.0000153

# one query, 20 candidates at that size
20 × 364 = 7,280 tokens × $0.042 / 1M = $0.000306

# 100,000 queries a month, 20 candidates each
2,000,000 × 364 = 728,000,000 tokens × $0.042 / 1M = $30.58

Watch the request limit before the bill. At 80 requests a second, one call per passage caps you at four twenty-candidate queries a second. Putting several passages into one state with a Score question for each cuts the request count, at the price of every question seeing the other passages as noise — measure both on your own data.

When not to use Jev for this

• You rerank hundreds of candidates per query. Requests and cost grow with the set; narrow it with cheaper retrieval or metadata filters first.
• Relevance turns on numbers or dates — which price applies after a given day, whether a limit is above a threshold. Jev compares those unreliably; filter on metadata in code.
• Passages are whole documents. The state plus the rubric must fit in 32k tokens, and long passages dilute the judgement well before that limit.
• Users need to see why a source was chosen. Jev returns a score, not a reason.

Where it fits in your stack

Jev reranking for RAG belongs between the vector store and the generator: retrieve generously for recall, score every candidate, keep those at or above your band, and hand the survivors to the LLM in score order. Embeddings stay in charge of recall; Jev decides which of the recalled passages earn a place in the context.

Cache scores on the question-and-passage pair — the measured pair scored between 0.93 and 0.97 across five runs, so a repeat call buys little. Then log dropped passages next to a rating of the final answer: if good answers keep needing passages you dropped, the cut point is too high. typesafe-jev-examples has a reranking example that runs through OpenRouter.

Notes from the field

• Passage classification appears among TypeSafe's published cookbook examples.
• Add a Noul for "this passage contradicts the question's premise" to catch a failure mode similarity scoring is blind to.
• Run the calls for one query concurrently — each passage is scored independently, so nothing forces them into sequence.

Common questions