Evidence
A decision model in a 96-game benchmark
TextArena is the standard suite for measuring language models on games. Jev can enter ten of them, and beats a bot you could write on one line in two. This is the result, including the part where it loses.
96 games, 10 it can enter
TextArena's agent contract is step(action: str) — you hand it a string. Jev generates no text at all, so it cannot satisfy that contract on its own terms. Where a game prints its own legal moves, the option set comes free and a Choice can pick one. Where it does not, someone has to write a candidate generator, and then the benchmark measures the generator.
16 of the 96 families print their moves at reset, and 10 produce a completed game. 3 stop enumerating partway (Golf, MarketEntryGame, PublicGoodsGame) and 3 never finish inside the turn cap (SpiteAndMalice, Stratego, SantoriniBaseFixed). A scan gives you candidates; only running them gives you the number.
8 of those 10 are in the table below. Chess and Othello are left out and it is worth saying why rather than quietly dropping them: Chess made 761 truncated calls and 7 of its 10 games hit the turn cap, so its score would be a finite-context agent's rather than the model's, and Othello abandoned 8 of 10 games at a forced pass, leaving two games of data, which is none.
Two wins, one loss, five ties
| Game | Jev | One-line bot | Difference | Jev's confidence |
|---|---|---|---|---|
| Crusade | 1.00 | 0.28 | +0.72 | 0.28 |
| Snake | 0.48 | 0.17 | +0.32 | 0.14 |
| Indian Poker | 0.71 | 0.97 | -0.26 | 0.41 |
| Surround | 0.62 | 0.49 | +0.13 · | 0.15 |
| Wild Tic Tac Toe | 0.71 | 0.81 | -0.10 · | 0.40 |
| Reverse Tic Tac Toe | 0.42 | 0.36 | +0.05 · | 0.28 |
| Tic Tac Toe | 0.64 | 0.64 | +0.01 · | 0.41 |
| Simple Tak | 0.70 | 0.71 | -0.01 · | 0.29 |
A trailing · marks a difference inside two standard errors — at 100 games each that is ±0.14, so those five rows say the two players are indistinguishable, not that they are equal.
Crusade and Snake: it really is better
1.00 against 0.28, and 0.48 against 0.17. Both far outside the noise. Crusade in particular is a clean sweep — and it is not the fallback in disguise, because 0 of 320 calls there returned a choice outside the legal list.
Indian Poker: it is worse
0.71 against 0.97. The one-line bot nearly always wins that game, and Jev does not. Publishing this is the point: a benchmark that only reports the two wins is an advertisement.
The part that holds up everywhere
Confidence sits between 0.14 and 0.41 across every game — consistently low, and lowest where lookahead matters most. Jev is not good at board games and appears to know it. That is not a result a chat model can even report: they return no confidence at all.
Reproduce it
bench/arena.py in jev-measured. Bring a key and re-run it; disagree with a row and you can open an issue against a specific number.