Jev tool selection: Jev picks the tool, an LLM writes the arguments

The problem

The standard agent loop stuffs every tool schema into one prompt and asks the LLM to choose and call. Cost and latency grow with the tool count, and accuracy degrades once you are past a few dozen tools.

How Jev handles it

Separate the two jobs. Selection is a classification problem over a known option set — exactly Choice. Argument-writing genuinely needs generation, so keep an LLM for that step only.

# 1. Jev selects — criteria is a dict of option -> when to pick it,
#    so tool descriptions become the selection signal.
response = client.system_one(
    model="jev-latest",
    state=conversation,
    questions={
        "tool": Choice(
            instructions="Which tool should run next?",
            criteria={t.name: t.description for t in available_tools},
        ),
    },
)

# 2. An LLM writes arguments only for the winning tool
answer = response.answers["tool"]
if answer.confidence >= THRESHOLD:
    tool = registry[answer.choice]
    return tool.run(**llm.fill_arguments(tool.schema, conversation))
return llm.full_agent_step(conversation)

Does it actually work?

Writing tool descriptions for Jev tool selection

With Jev tool selection, each tool's description is the entire signal, so write it as a trigger — "find available flights between airports on a date" — rather than a summary of what the code does. Tools that sit next to each other need descriptions that split the difference out loud. The measured fixture separates search_flights from book_flight with "purchase a specific flight already chosen", and in the measured run book_flight received no probability at all.

Two options belong in almost every toolset: one for answering directly with no tool, and one for asking the user a clarifying question. Without them Choice has to pick a tool even when none applies, because it only chooses among the options you send. It also returns one winner per call. The measured request needed two steps and Jev chose search_flights first, at 0.99 confidence; the hotel email is the next call's decision, once the flight result is back in the state.

What to put in the state

In Jev tool selection the state is whatever the agent is acting on: the user's latest request, the tool results so far, and a short record of steps already taken ("DONE: search_flights, 3 options returned"). That record is what stops the same tool winning twice in a row. The tool list does not go in the state — it lives in the Choice criteria, which count against the 64k request budget, and the state plus that one question must also fit inside the tighter 32k limit.

Long transcripts are the usual failure. Window them to the last few turns or summarise the older ones, because an old tangent left in the state competes with the current request.

What one decision costs

The measured Jev tool selection call carried five tools. Production agents carry more, so the second line assumes 100 tools at about 20 tokens of description each plus a 1,500-token conversation: 3,500 input tokens per step, an assumption to replace with your own count. The third prices a ten-step task at that size.

# the measured call, five tools
403 tokens × $0.042 / 1M = $0.0000169

# one step with 100 tools and a 1,500-token conversation
3,500 tokens × $0.042 / 1M = $0.000147

# a ten-step task at that size
10 × 3,500 = 35,000 tokens × $0.042 / 1M = $0.00147

The saving shows up on the LLM side: once Jev has chosen, the argument-writing prompt carries one tool schema instead of a hundred.

When not to use Jev for this

• You have more than 255 tools. That is Choice's ceiling — ask for the category first and the tool second, or pre-filter with retrieval.
• Picking the step needs a date or number comparison — is the flight before the meeting, is the balance enough. Do that in code and put the answer in the state.
• Several tools should run at once. Choice returns one winner. For parallel calls, ask one Noul per tool ("should this run now?") and threshold each on its own, since a Noul and a Choice probability are not interchangeable.
• Your agent fails on arguments, not on tool choice. Jev never writes arguments, so it cannot fix malformed parameters.

Where it fits in your stack

With Jev tool selection the loop you already run barely changes: the framework asks Jev which tool is next, then asks the LLM to fill in arguments for that one tool only. When confidence falls below your threshold, fall back to the ordinary full LLM step, so the worst case is the agent you had before.

Public projects show the shapes this takes. jev-eval-agent compares step counts across 100 mocked tools; jev-ultrafast rebuilds the action space from the live page on every step and calls a small LLM only when text must be typed; and jev-mcp exposes Jev judgements as MCP tools another agent can call.

Notes from the field

• Build the options list fresh each turn from live state — a dynamic action space is the normal case, not an edge case.
• Only the winning tool's schema goes into the LLM prompt, which shrinks that prompt dramatically.
• Measure your confidence threshold against labelled traces; do not ship a guessed number.

Common questions