Building a Codenames Teammate with Jev

TypeSafe's Jev is a "System One" model. You give it context and typed questions. It returns decisions with probabilities instead of generating text.

Kiriti wrote a good introduction in What would you build with a much faster classifier model?. It is a great starting point for someone who wants to understand what Jev is and does.

What I liked was the interface. In quite a few scenarios, all an application needs from a model is a judgment about a fixed set of options. An opiniated number for each option gives the code something for it to act on. I wanted to see what building with it feels like.

So I built a Codenames teammate. It was a small experiment: connect a clue to words on a board, then watch the app act on Jev's answers. The integration was pretty easy, and it took one request per clue. This also gave me an opportunity to experiment with the different types of questions to ask.

So, What is Codenames?

Codenames is a word-association game. There are 25 words on a board. Each word secretly belongs to the red team, the blue team, neither team, or the assassin. The goal is to find your team's words without revealing the assassin.

The two roles in the game:

• The spymaster ... sees the hidden colors and gives clues.
• The guesser ... sees only the words and tries to figure out what the spymaster means.

In this app:

• I am the red spymaster.
• Jev is the guesser.

This is my view of the board:

As an example: Brush and Paste are both red. I want my teammate to choose them without actually saying their names. So, instead, I give the clue "dental, 2."

"Dental" is the connection. The number 2 says how many words I mean. It is part of the clue, not a confidence score.

My teammate chooses a word, and the game reveals its color:

• Picking a red word is a correct guess. You can continue to pick.
• Picking a blue word or a neutral word helps the other team and you lose your turn.
• Picking the assassin means you lose the game.

For this app, I limit the guesses for each clue. With "dental, 2," the intended turn is Paste, then Brush, then stop.

Jev's Job Is to Connect the Words

The app has been built to reveal a card and check its color. What it needs from Jev is a judgment: which words on this board does "dental" point to?

I send the clue, it's count, and the remaining words in one request. Jev's job is to make the same association a human guesser would, without seeing the answer key (i.e., the colors).

This is why I wanted to try Jev here. It can return a judgment for each candidate, and the app can act on those answers directly. There isn't a need for any prose/explanation to parse.

But, first things first, I had to decide what kind of judgment to ask for.

Choice vs. Noul

I started with Jev's Choice question type. It asks, "Which of these options?" and returns a probability for every option. The thing to remember is that these probabilities sum to 100%.

That sounded like the obvious fit: Which board word does this clue point to? But a clue can point to more than one word.

Noul asks a yes-or-no question and returns the probability of yes. For the Dental clue, I can ask: Is Paste one of the two words the clue points to? Then, ask the same question about Brush, and then for every other unrevealed word. All those questions fit in one request.

The percentages below are Jev's answers to those questions. They are model outputs, not part of the game's rules.

In a prior test on a different board, I compared both question types with "Animal, 2." Here are four words from that board:

Word	Choice	Noul
Rabbit	61%	93%
Bear	24%	84%
Scorpion (assassin)	12%	80%
Crane	3%	64%

For "animal, 2," both question types ranked Rabbit first and Bear second. The probabilities mean different things.

With Choice, Bear at 24% feels like a distant runner-up. With Noul, Bear at 84% is a strong second word. Choice makes the words share a total probability of 100%. Rabbit's 61% leaves at most 39% for every other word combined, however well bear fits the clue.

Noul evaluates each word on its own terms. Bear can be a strong candidate without making Rabbit a weak candidate.

That fits the game better. I needed to find several words that belong with the clue, so I switched to Noul.

In my experiments, the 25 Nouls returned in roughly the same 200 milliseconds as one choice.

Turning the Answers into Guesses

Back to "Dental, 2" on the screenshot board. With Noul, Jev returned Paste at 92%, Brush at 91%, and Nail at 65%.

Now the app needed a policy for making use of these answers. The logic was simple ... choose the highest-scoring words first, without exceeding the clue count, and skip anything below 50%. That cutoff is an implementation choice.

The app revealed Paste, then Brush. Both were red. It stopped after those two guesses, even though Nail also cleared the cutoff.

The Code

So, what are the the two things to implement?

1. Ask Jev how well each word fits

The model call runs server side, using TypeSafe's SDK. The quick start covers installation and client setup. candidates contains only the unrevealed words, without their hidden colors:

const r = await client.systemOne({
  state: { game: 'Codenames', clue, count, board: candidates },
  questions: Object.fromEntries(
    candidates.map((word) => [
      word,
      noul(
        `Is the board word "${word}" one of the ${count} words the spymaster's clue "${clue}" points to?`,
      ),
    ]),
  ),
});

const probabilities = Object.fromEntries(
  candidates.map((word) => [word, r.answers[word].noul]),
);

2. Apply the game's rules to the returned answers

The app then sorts the probabilities and selects up to count words above the cutoff:

const ranked = Object.entries(probabilities).sort((a, b) => b[1] - a[1]);

const guesses = ranked
  .slice(0, count)
  .filter(([, probability]) => probability >= 0.5)
  .map(([word]) => word);

This produces the list of intended guesses. The game reveals them in order and ends the turn if a guess is wrong.