Jev Wikirace: watch a decision model race Wikipedia

What happens on each Jev Wikirace hop

Each hop, your browser sends the current article, the target and the visited list. The server does the rest:

1. It asks the Wikipedia API for the current article's links — up to 500 in one request, articles only, so File:, Talk: and Category: pages never appear.
2. If the target is among them, it takes that hop itself. No call is made, and the hop is tagged “target was linked”.
3. Otherwise it builds the shortlist. Visited articles, hub pages such as Main Page, and every “List of” article are dropped. Titles containing a word of the target's name longer than three letters go first; the remaining slots, up to 48, are sampled evenly from everything else.
4. Jev gets one Choice question — which of these links leads closest to the target — and its pick becomes the next article.

One consequence is easy to miss. Because a linked target is taken before Jev is asked, the target is never one of its options: its real job is to reach any page that links there. The browser allows eight hops. Each one is paid for by this site, so one address can make 20 a minute, and all visitors together 3,000 a day.

What Jev sees, and what it never sees

This is the whole of the first hop of the Coffee preset, as Jev receives it:

state:
  You are playing Wikiracing.
  Current article: Coffee
  Target article: Napoleon
  Already visited: Coffee

choice:
  Which of these links leads closest to the article "Napoleon"?
  options: the shortlisted titles, each described by its own title

Nothing else. In a Jev Wikirace the model never reads the article it is standing on and has no map of Wikipedia's link graph. Each hop is a judgement about which title sounds closest to the target, from what the model already associates with those words — and it learns whether the bet was good only when the next page's links arrive.

Reading the numbers under each hop

• The bars are Jev's five highest-rated options. A flat top five is the model telling you it is guessing.
• Links on page and sent to Jev: the gap between them is the part of the decision made by code.
• Latency covers the model call alone, clocked by the server; the Wikipedia lookup is not in it.
• Cost is what the upstream route reported, or a dash where it reports none.

For scale: the one 48-option Wikirace hop measured for the pricing page came to about 1,100 input tokens and $0.0000474. Output is priced at zero, so input is the whole bill.

Why a game is the honest demo

A game is a long run of decisions over an action space that changes every step. That is the exact shape Jev exists for, which is why TypeSafe reached for games in their own launch post rather than a benchmark table.

It is also unusually hard to fake. You can see every option it was offered, the probability it gave each one, and what the move cost — all in the same view.

TypeSafe's own two demos

Doom is the more spectacular of the two but the harder one to host: at roughly ten queries a second it runs about $7/hour, which is why the demo on this page is a Wikirace. Note also that Doom reads a text state, not pixels — text only — no image or audio input at launch

The 255-option ceiling

A Wikipedia article can link to hundreds or thousands of others, and Choice accepts at most 255 options. TypeSafe handle this with a two-stage system. This page takes the cheaper route and shortlists 48 links per hop — the options are the prompt, so every extra link is input tokens on every move.

Which is the real lesson for anyone building on a decisions model: the candidate set is yours, and it is where your bugs will live. Jev can only be as good as the options you hand it.

What a Jev Wikirace proves, and what it does not

It proves the mechanics. A decision model can take an option set that is different on every step and return a distribution over it in one call, at a price you can read to eight decimal places. Nothing in the loop needs generated text: the browser keeps the path, the server keeps the rules, and Jev only ranks.

It does not prove Jev is a good Wikiracer. A two-hop win can come from a lucky shortlist, and this page runs no baseline — the only comparison is TypeSafe's, in the card above. Where we did run a control, in the game benchmark, Jev beat a one-line bot at 2 of 8 games and lost at 1. For an agent, the part that transfers is the loop: code lists the moves, Jev ranks them, code acts.

Jev Wikirace FAQ

Article data from the Wikipedia API. Demos referenced from TypeSafe's launch post.