Jev game benchmark: a decision model in TextArena's 96 games

96 games, 10 it can enter

TextArena's agent contract is step(action: str) — you hand it a string. Jev generates no text at all, so it cannot satisfy that contract on its own terms. Where a game prints its own legal moves, the option set comes free and a Choice can pick one. Where it does not, someone has to write a candidate generator, and then the benchmark measures the generator.

16 of the 96 families print their moves at reset, and 10 produce a completed game. 3 stop enumerating partway (Golf, MarketEntryGame, PublicGoodsGame) and 3 never finish inside the turn cap (SpiteAndMalice, Stratego, SantoriniBaseFixed). A scan gives you candidates; only running them gives you the number.

8 of those 10 are in the table below. Chess and Othello are left out and it is worth saying why rather than quietly dropping them: Chess made 761 truncated calls and 7 of its 10 games hit the turn cap, so its score would be a finite-context agent's rather than the model's, and Othello abandoned 8 of 10 games at a forced pass, leaving two games of data, which is none.

How the Jev game benchmark asks for a move

Every turn, the harness reads the last line of the game's own output that lists the available moves and turns each move into an option. The question never changes: you are the player to move — which of these moves gives you the best chance of winning? The state is the game's observation as TextArena printed it.

Two things make that harder than it sounds. A move such as [4] is described only as “Play [4]”, and since option descriptions are the whole prompt surface of a Choice, this is close to the worst case for how Jev is built. And observations grow turn by turn, so the state is cut to its last 12,000 characters, where the current board and move list sit. Each row records how often that fired: 506 calls in Crusade, and none in any other scored game.

If an answer is not one of the listed moves, the harness plays the first legal move instead — the baseline's own strategy, which is why the FAQ below comes back to it. Games alternate seats, run to a 200-turn cap, and a game stopped by the cap is not scored as a draw; draws that do finish count as half a win.

Jev game benchmark results: two wins, one loss, five ties

A trailing · marks a difference inside two standard errors — at 100 games each that is ±0.14, so those five rows say the two players are indistinguishable, not that they are equal.

Crusade and Snake: it really is better

1.00 against 0.28, and 0.48 against 0.17. Both far outside the noise. Crusade in particular is a clean sweep — and it is not the fallback in disguise, because 0 of 320 calls there returned a choice outside the legal list.

Indian Poker: it is worse

0.71 against 0.97. The one-line bot nearly always wins that game, and Jev does not. Publishing this is the point: a benchmark that only reports the two wins is an advertisement.

The part that holds up everywhere

Confidence sits between 0.14 and 0.41 across every game — consistently low, and lowest where lookahead matters most. Jev is not good at board games and appears to know it. That is not a result a chat model can even report: they return no confidence at all.

About half a second a move

Median latency per decision in the Jev game benchmark sat between 504 and 537 ms in every scored game, timed around each API call from the machine running the benchmark — a round trip, not server time. It barely moves with the result: Crusade, the clean sweep, took 537 ms a move and Indian Poker, the loss, 533 ms.

What the Jev game benchmark does not prove

It is 8 games, one opponent and one prompt. The opponent is deliberately weak — the point is a floor, not a rating — so a tie says Jev did no better than playing the first move on the list, not that it plays the game well. It says nothing about games whose moves must be generated, which is most of TextArena, and nothing about chess, left out for the reasons above.

There is no chat model in the table either. The harness can seat one as the opponent, held to the same move list by a JSON schema at temperature 0, but no such run is published, so this page makes no claim about Jev against an LLM at games. For that comparison on decision tasks, see Jev vs an LLM and the benchmarks.

Reproduce it

The Jev game benchmark harness is bench/arena.py: it needs the textarena package and a TypeSafe or OpenRouter key, and the method section above describes what it does. It is not yet in the public jev-measured repository, which holds the cost, accuracy and head-to-head runs; until it is, challenge a row at support@jev-agent.com.

Jev game benchmark FAQ