This is an experiment where a large language model (LLM) named Claude plays through the classic Game Boy game Pokémon Red, entirely on its own. Unlike traditional game bots that follow predetermined rules, Claude can see what's happening, understand the game state, and make decisions, similar to how a human player would.
This project is a passion project made by a person who loves Claude and loves Pokémon.
As Anthropic releases new models, we'll kick off new runs and see how they do -- one of these days Claude will be a Pokemon Master!
Throughout the stream, you'll see Claude encounter challenges, develop strategies, raise Pokémon, battle trainers, and make progress toward becoming a Pokémon Champion. You can watch its thought process in real-time, showing how an AI approaches problem-solving in an interactive environment.
Claude can be a little silly sometimes – tune in and enjoy the show!
The system combines several components that enable Claude to play Pokémon:
Game Interface: A custom interface that allows Claude to control the game by pressing virtual buttons.
Screenshot Analysis: Claude looks at screenshots of the game and interprets what's happening on screen. Claude's ability to understand Game Boy screens isn't great—you'll often see it misinterpret what it's seeing.
Knowledge Base: Claude maintains a dynamic set of notes about the game world, storing information about locations, Pokémon team status, and game mechanics.
Navigation System: A pathfinding tool helps Claude navigate efficiently by finding paths to a spot on the screen.
Memory Reader: Claude also gets a few tidbits of information from the memory of the game, like its current location and party. This helps prevent Claude from getting confused if it misinterprets the screen.
When Claude wants to act, it can:
Note: We're using Danny-E 33 + FroggestSpirit's full color patch, which also helps Claude see the screen a bit better!
