Language models vs. the daily word puzzle

Models

About

Compares language models' ability to solve real NYT Connections puzzles with the official rules. Every guess is logged for comparison, along with the LLM's reasoning.

Why this exists

I play Connections almost daily along with other NYT Games. The game is simple with easy to describe rules but often tests your creativity and ability to think outside the box. I know LLMs know more information than me, as they are training on the totality of human writing. Connections is about more than just knowing things though, and I wanted to know whether or not an LLM could reliably solve the puzzles.

Why this is a useful benchmark

Connections demands lateral thinking and cultural context. Puzzles may utilize wordplay, idioms, niche knowledge, and soft associations to trick the player. That stresses the breadth and depth of a model’s training in a way common evaluation suites may not.

Models play the game like a human would - four mistake limit, multi-turn play (not one-shot classification), one puzzle per chat context, and full guess-plus-reasoning logs. Runs go through OpenRouter with a shared prompt so each day's results are comparable.

How it works

  1. The model is told how to play and given the current game board.
  2. It guesses a four-word category. The engine replies whether the guess is correct, incorrect, or invalid (for example, wrong word count or already-solved words).
  3. Play continues until the puzzle is solved or the model runs out of mistakes.

Each turn also records the model’s reasoning, which you can open on any date page’s guess-by-guess replay.

How to read the results

  • Season standings rank models by win rate (solved ÷ completed games), then average mistakes. You’ll also see W–L, cost, tokens, and games played.
  • Mistakes are wrong category guesses (four lose the game). Invalid guesses break the rules and don’t count as mistakes.
  • Date pages show the board, outcomes per model, and expandable guess history with reasoning.
  • The model filter defaults to just a few major models, but I've run the benchmarks through many available models. Open the filter to include more models in the result set.
  • Note: some models are too expensive or too slow to run through a backlog of puzzles. Feel free to steal the code and run it yourself if you’re curious.

Limitations

  • Unofficial project — not affiliated with The New York Times or Connections.
  • Puzzle archive comes from a third-party source (Eyefyre/NYT-Connections-Answers), not NYT directly.
  • Older puzzles may appear in training data for newer LLMs.
  • There are a few gaps in the game model. It does not communicate the difficult of a given category when a match is found. It also does not share "1 away" information when the LLM's guess is nearly correct.
  • The harness mostly uses OpenRouter model defaults with regards to thinking effort. Tokens/game can be an indicator of these different effort levels.

Author & links

Built by Steve Freund. Source code is on GitHub. Models are called through OpenRouter; puzzles from Eyefyre/NYT-Connections-Answers.