Jev LLM router: pick the cheap model first, escalate the rest

The problem

Model cascades only pay off if the router is cheaper and faster than the models it routes to. Using an LLM as the router eats most of the savings — you have added a full round-trip to save a full round-trip.

How Jev handles it

A Jev LLM router classifies difficulty or task type in a fraction of the time, so the router cost rounds to zero against the model call it selects. This is arguably the single most natural fit for a System One model.

response = client.system_one(
    model="jev-latest",
    state=user_prompt,
    questions={
        "tier": Choice(
            instructions="How much capability does answering this require?",
            criteria={
                "trivial":         "Lookup, greeting, or one-line factual answer",
                "standard":        "Ordinary request, no multi-step reasoning",
                "complex":         "Multi-step or domain-specific work",
                "needs_reasoning": "Requires planning, math, or careful analysis",
            },
        ),
    },
)

MODEL = {
    "trivial":         "small-fast-model",
    "standard":        "mid-tier-model",
    "complex":         "frontier-model",
    "needs_reasoning": "frontier-reasoning-model",
}[response.answers["tier"].choice]

Does it actually work?

Designing the tiers for a Jev LLM router

Write one option per model you actually run, and describe each tier by what that model reliably handles rather than by how hard the prompt feels. "Trivial" only earns its place if it means "the small model gets this right". If your cheap model also copes with one-step code edits, say so in its description: the descriptions are the only definition of the tiers that a Jev LLM router ever sees.

Keep countable features out of the question. Prompt length, number of attachments and conversation depth are facts your code already has, and counting is on TypeSafe's own list of things Jev does not do reliably — route on them before the call, or state them as plain text. Extra dimensions belong in extra questions rather than longer option text: a Noul asking whether the prompt needs current information can send it to a search-enabled model whatever its tier, and it shares the call.

What to put in the state

Route on the newest user message, plus the previous turn when the message only makes sense as a follow-up ("do the same for March"). Leave out the system prompt and the long conversation history: the router judges the request, and every extra paragraph is both a distractor and a cost paid on every call.

The measured state was five words, and the call still billed 369 input tokens. That shows where a short prompt's cost really sits — in the question definition — and it is why tier descriptions should stay tight: they are resent with every prompt you route.

What one decision costs

A Jev LLM router is paid on every request, including those that end up at the frontier model anyway, so price it per routed prompt. First line: the measured call. Second: an assumed 2,000-token prompt that carries recent context. Third: a million routed prompts a month at the measured size.

# the measured prompt, one Choice
369 tokens × $0.042 / 1M = $0.0000155

# a 2,000-token prompt with recent context
2,000 tokens × $0.042 / 1M = $0.0000840

# 1,000,000 routed prompts a month at the measured size
1,000,000 × 369 = 369,000,000 tokens × $0.042 / 1M = $15.50

Set that against the price gap between your cheapest and dearest tier on every prompt the router keeps away from the expensive model. When the gap is small, or nearly all traffic genuinely needs the top tier, the router is overhead rather than saving.

When not to use Jev for this

• Almost every prompt needs your strongest model. A router in front of a one-tier cascade only adds a hop.
• Your fast path is already fast. Through the first-party API the measured routing call took 270 ms median end to end; through OpenRouter, 458 ms from our machine. If the small model answers in a few hundred milliseconds, the routing hop can cost as much time as it saves.
• The rule is something code can compute — token count, file type, the user's plan. Do that in code; it is exact and costs nothing.
• Prompts include images or audio. Jev reads text only, so it cannot judge whether a screenshot needs a vision model.
• Each routing decision needs a written justification for an audit trail. Jev returns a distribution over tiers, never a reason.

Where it fits in your stack

The Jev LLM router lives in your gateway layer: a request arrives, one Choice call picks the tier, and the model call goes to the mapped endpoint. Gateways that already carry Jev — OpenRouter as typesafe/jev-1.13, Vercel AI Gateway as typesafe-ai/jev — keep both calls behind one account, and pi-jev-router adds exactly this per-request routing to the Pi coding agent on Vercel AI Gateway.

Two safety nets make it production-safe. A short timeout on the routing call that falls through to your default model, so a Jev outage costs money rather than availability. And an escalation path: when the cheap model's answer fails a check, resend one tier up and log the prompt, because those are exactly the cases the tier descriptions got wrong.

Notes from the field

• Route on your own tiers, not generic difficulty — "needs a tool" or "is a follow-up" often predicts cost better than "hard".
• Log the routing decision alongside the eventual outcome so you can tune the tier definitions against reality.
• Keep a straight-to-frontier fallback: if Jev is unavailable, the product should degrade in cost, not in function.

Common questions