A tiny world model running games on a Raspberry Pi Pico 2 W

Using 2 small neural networks to create and perform the equivalent at the identical period on a Raspberry Pi Pico 2 W. The archetypal one creates the game: stated the current example and the action, it predicts the next example and the reward. The second one plays that game: it uses the generated example and picks an action.

Every example complete was sampled by the earth example network, and the actions were taken by the performer network.

I got curious in looped transformers, anywhere one transformer obstacle is applied multiple times alongside the identical weights alternatively of stacking distinct layers. Reusing weights is cheap on memory, so I wanted to see if a looped earth example could be small adequate to run on a microcontroller. I picked the Pico 2 W since it is cheap and has an FPU, and small games since I could compose the genuine versions immediately into numpy and inspect all generated example against them. Later I added a streaming RL delegate so the commission could additionally study to perform inner its own dream.

Contents: the two networks · the games · the earth model · the agents · quantization · on the chip · flashing · browser · reproduce · code layout

The size depends on the game, several games necessitate additional parameters to study them to be playable, we tried to get to keep the figure of the parameters small enough.

On the part neither network always uses the genuine game: the performer lone sees frames the earth model, and the earth example lone gets actions from the actor. Both run on center 1 of the RP2350, during center 0 handles USB and the display.

The pretrained performer can be exchanged for a caller stream-x delegate that learns from scratch on the board. And any clasp media hands authority to you, so you can perform the aspiration yourself.

Each equivalent is a few hundred lines of numpy in src/tinyworldformer/games/. It renders to a small grid of token ids, and one example continually holds the entire equivalent state, so hidden province have to be drawn into the frame: the ball's heading is part of its token, and Doom-lite has a minimap.

"Exact frames" is the portion of held-out transitions anywhere all division of the predicted example matches the genuine game. Frames correct following a random event are remaining out, since those can lone be sampled, not predicted. Doom-lite's 3D perspective has 768 cells and a sole shaded partition pixel off counts the entire example as wrong, so its figure looks worse than it plays; the equivalent province is nearly continually correct (see checking the dream).

You can perform the genuine games in a terminal: uv run python -m tinyworldformer.games.play --game pacman (or breakout, doomlite, flappy, pong).

The example is cut into quadrate patches: 2×2 cells in Breakout, 4×4 in Doom-lite, Flappy Bird and Pong, 6×6 in Pac-Man. That gives 36 to 80 tokens per game. A patch's embedding is the sum of one learned vector per cell, alongside a distinct array for all stance inner the place so the division command isn't lost. Every token additionally gets a stance vector and the action's vector.

The example is a sole transformer obstacle (attention affirmative a ReLU MLP) alongside one set of weights, applied 1 to 4 times, alongside the input added rear before all pass. The output following all continue is trained against the next frame, so at run period the iteration figure trades speed for accuracy. On the Pico to keep a decent example charge we use uses two loops.

The animation shows one held-out Doom-lite step. After one continue the example draws an enemy that isn't there, and all additional continue removes part of it: 92, 35, 12, afterward 0 incorrect cells. The bars on the correct are the portion of held-out frames all iteration figure gets exactly right.

Here is a dreamed Doom-lite stage decoded following 1, 2, 3 and 4 loops, alongside red boxes on cells that differ from the genuine rules:

Exact held-out frames by iteration count:

The example outputs a mark for all division and all token. Turning those scores into a example turned out to matter nearly as much as the model, and it ended up as three rules (shown complete on Pong):

1. Background cells (walls, bricks, pellets, floor) all obtain their most apt token independently.
2. Singletons (the ball, the player, all ghost) are placed alongside one choice complete all (cell, token) pairs, so there is continually exactly one of each. After a random event the choice is sampled among candidates inside 2.2 logits of the best, so the earth example samples a new assist or respawn alternatively of averaging them.
3. Pac-Man and the ghosts are 2×2 sprites, but lone one division of all carries province (the mouth holds the heading; the ghost's lower-left division holds whether a pellet is underneath). The another three cells are stamped from it alternatively of predicted.

We implemented Rule 2 since the archetypal Breakout example drew two balls at once; one earth choice took exact frames from 91.6% to 99.7%.

Accuracy on recorded data wasn't enough. An delegate drives the aspiration into states the data never had, and any error there is item it can exploit. So the chief inspect runs during an delegate plays: peruse the equivalent province rear off the dreamed example (games/<game>.py: read), use the genuine rules to that province and the agent's action, and difference the outcome alongside the next dreamed example and reward.

5,000 dreamed steps per equivalent at 2 loops, alongside the shipped delegate (training/evalwm.py --agent):

First, agents perform the genuine equivalent and all stage is recorded as one example (frame, action, next frame, reward), 1–3 myriad per game. Some episodes commencement from different states, akin bricks alongside holes or random positions, so the data covers additional of the game.

Then the earth example learns to foretell the next example alongside a per-cell cross-entropy loss: 30k steps, alongside the archetypal 3–4k steps run in float, afterward int8 rounding switches on (QAT).

The int8 weights are written into Rust origin alongside all drift stored as its exact bits. We inspect them alongside cargo test to reproduces the Python entire figure spec bit for bit, and afterward the firmware runs on the Pico.

An delegate trained in the earth example reaches states the recorded data never had, so document genuine games alongside that delegate and fine-tune the earth example on everything.

The delegate in distinct games stated exploiting them so those are several fixes to it:

• Weighting the defeat by "this division changed" biased the example toward predicting change, so it drew two balls and bankrupt two bricks. Weighting by anywhere things can happen in the input frame, affirmative the earth singleton choice, fixed it.
• A Breakout delegate in the aspiration got 154 points alongside 26 bricks left, since the earth example didn't eliminate bricks from the border whenever it were hit. Doom-lite agents learned to snipe from 5 cells away.
• Agents trained lone in the aspiration scored 210 there (the genuine maximum is 56) and 28 in the genuine game, as World Models (Ha & Schmidhuber) predicts. Each collect-and-fine-tune circular closed the utilize that delegate had found.
• The Doom-lite aspiration sometimes misplaced the enemy sprite and never drew it again: the example copied the former perspective alternatively of redrawing it from the minimap. Noise augmentation fixed it (train --noise 0.3, as in GameNGen): input views get random pixels and sometimes endure the sprite.
• d=32 couldn't assimilate the wider Breakout data and d=48 could. Pac-Man's 48×48 example needed d=64.

The agents use Stream AC(λ) (Elsayed, Vasan & Mahmood, 2024), which learns from one passage at a time: no replay buffer, no batches, no mark network. Eligibility traces transport credit rear in time, and ObGD shrinks any stage that would overshoot the TD target. The guideline and value networks all have two hidden layers of 64 units (32 on the Pico) alongside LayerNorm.

The input is a characteristic vector peruse off the example (training/streamx.py: observe_*, mirrored in Rust in tinyworldformer-rs/src/agent.rs). For example, the ball's row, pillar and heading in Breakout, or in Pac-Man, for all direction, the shortest-path extend to the nearest pellet and how near all shade is.

Below is one training run per equivalent (real game, 500k steps) at six stages: untrained, the end, and the snapshots anywhere the mark has covered 25, 50, 75 and 95% of the way. Each panel plays greedily from the identical commencement and skips a few steps per frame. "try" counts the episodes played so far, so you can see the crashes.

Breakout

Doom-lite

Flappy Bird

Pac-Man lite

Pong

Greedy come back in the genuine game, average of 10 episodes, at snapshots of the identical run:

Without shaping, Doom-lite agents campsite in the commencement border and fire, since the enemy continually comes to them. --idle-penalty 0.1 expenses 0.1 for all stage spent in a division visited additional than 6 times in the final 20 steps. The stance is peruse off the minimap, so the penalty plant inner the aspiration too. With it the delegate patrols and gets twice the kills.

tinyworldformer-rs/src/learner.rs is the identical algorithm in no_std Rust, alongside the backward continue written by hand. It matches 200 PyTorch updates to inside 3.5e-6. On the host, a caller delegate inner the Rust int8 Flappy Bird aspiration averages 6 pipes per event following 25k steps, 121 following 100k, and 300+ following 250k.

Every matrix in the earth example is int8 alongside one measure per output row (scale = max|w| / 127). Activations are int8 too, alongside one measure per tensor and per iteration index, since the loops see distinct ranges. Matmuls sum int8 × int8 into int32 (two multiply-adds per SMLAD education on the RP2350) and afterward use one drift multiply by the two scales. Softmax uses a 1024-entry exp table, alongside probabilities stored as uint8. Norms, residuals and embeddings remain in float32, since the RP2350 has an FPU.

Training applies the identical rounding (QAT alongside a straight-through estimator), so the int8 example is the example that was trained. rustexport/intref.py is the int8 example in numpy alongside all drift procedure in a fixed order; the Rust motor reproduces it bit for bit, and cargo test checks aureate vectors for all exported env.

Flash and RAM are measured from the firmware ELF (size -A), built alongside the Display Pack driver, so they contain the code, USB, the display controller and the on-device learner. The Pico 2 W has 4 MB of flash and 520 KB of SRAM, 512 KiB of it in the chief recollection map. The float32 embedding tables are the largest part and could be int8 too.

src/tinyworldformer/rustexport/ imports lone numpy and torch (a test keeps it that way), so another projects can reuse the export: build a example from rustexport.qat (QLinear, LoopedBlock), train it alongside ActQuant.enabled = True following a drift warm-up, change it alongside rustexport.convert.looped_model, afterward compose the Rust origin alongside int_model_arrays, mlp_arrays and codegen.write_module. rustexport.Golden writes the vectors a Rust test checks.

The firmware (tinyworldformer-rs/firmware) is no_std Rust on embassy. Nothing is allocated at run time: the weights are fixed arrays in flash, and all buffer is a fixed-size fixed in RAM.

Core 1 runs the game. One continue of its iteration (dream_loop in firmware/src/main.rs) is one frame:

1. The performer says features off the current example (agent::observe) and runs its MLP to choice a button. If on-device learning is on, the learning delegate picks instead; if you are pressing buttons, your media is used.
2. The int8 transformer runs 2 loops complete the example and the clasp (Engine::forward), and decoding turns its scores into the next frame: backdrop argmax per cell, all singleton placed formerly (sampled for random events alongside an xorshift RNG), afterward the sprite stamps. The reward is the argmax of the reward head; lives and event ends are counters on top.
3. If learning is on, the learner updates from this one passage and afterward forgets it.
4. The example goes to center 0 through a two-slot channel. If center 0 is motionless occupied alongside the former frame, this one is dropped, so center 1 never waits for the screen.

When you play, center 1 slows downward to a playable speed (66–150 ms per frame, depending on the game). The delegate runs as accelerated as the part allows.

Core 0 runs embassy async tasks for everything else. Over USB serial, all example goes out as coloured ANSI content alongside a position row (model time, performer time, fps, env and delegate names), and keys arrive rear in. The optional Display Pack (--features display) is a 320×240 ST7789 complete SPI at up to 62.5 MHz, and lone cells that changed since the final example are redrawn. Its buttons are polled all 20 ms. The two cores portion lone the example conduit and a few atomics: who is playing, the pressed key, learning on/off, and the iteration count.

Where the recollection goes:

Every equivalent fits in the 4 MB of flash and 512 KiB of SRAM. Breakout uses 361 KB of flash and 299 KB of SRAM; Pac-Man, the largest, uses 600 KB and 460 KB.

Most of the period goes into int8 dot products (dot_i8 in src/lib.rs). One 32-bit burden fetches 4 weights, SXTB16 unpacks bytes 0/2 and 1/3 into 16-bit halves, and SMLAD does two multiply-adds in one instruction. Attention multiplies uint8 probabilities by int8 values, so it uses UXTB16 instead. Norms, the exp array lookup and the final per-output measure remain in float32 on the FPU. A example at 2 loops is 2–4M multiply-adds (8.2M for Pac-Man); on a desktop CPU the identical Rust code dreams a example in 0.7 ms (Flappy Bird) to 1.5 ms (Breakout). I haven't measured the example charge on the Pico itself yet.

The commission boots alongside a caller delegate learning on the device. Press l complete USB, or clasp B for 1 s on the Display Pack, to toggle to the pretrained actor, and again to toggle back.

Every export sits flank by flank in the crate: earth models in tinyworldformer-rs/models/envs/<env>/, agents in models/agents/<game>/<agent>/. You choice the two at build time.

cd tinyworldformer-rs ./flash.sh --list # what is exported ./flash.sh --env doomlite --agent doom_agent_roam_p0.3 # build and flash (hold BOOTSEL during plugging in) ./flash.sh --env breakout --display # alongside the Display Pack ./flash.sh --env pong --host # perform it in this terminal ./flash.sh --env flappy --default # create it the default for plain cargo TWF_ENV=doomlite TWF_AGENT=doom_agent_roam_p0.1 cargo test --release # aureate tests for that pair

Without --agent, the delegate for the env's equivalent listed in models/default is used if there is one, alternatively the archetypal by name. An delegate for a distinct equivalent is refused alongside a catalog of the valid ones. The position row on display shows which env and delegate are running.

Keys complete USB: Breakout a/d/s, Doom-lite w/a/d/space/s, Flappy space, Pac-Man w/a/s/d, Pong w/s. Then g hands rear to the agent, 1–4 set the iteration count, l toggles on-device learning and r restarts it.

The identical no_std int8 motor additionally compiles to WebAssembly (tinyworldformer-rs/web/). The component has no imports and no wasm-bindgen, and it runs the firmware's loop: dream, agent, and stream-x learning on the fly. A fixed leaf draws the frames on a canvas.

rustup mark add wasm32-unknown-unknown # once cd tinyworldformer-rs/web ./build.sh # one component per env into www/ (or ./build.sh pong flappy) python3 -m http.server -d www # open http://localhost:8000, ?env=pacman picks a game

With the identical seed, the wasm build dreams the identical episodes as the native build, example for frame, at 1–5 ms per example depending on the game. Each component carries its game's default agent; TWF_AGENT=... ./build.sh <env> picks another. The keys are the identical as on the Pico, affirmative arrow keys and on-screen buttons, and f runs the aspiration as accelerated as it can, which helps during learning (l).

uv sync G=pong # breakout, doomlite, flappy, pacman, pong uv run python -m tinyworldformer.training.data --game $G --transitions 2000000 --varied 0.3 # -> data/$G uv run python -m tinyworldformer.training.train --data data/$G --d 48 --heads 3 --ff 96 --out models/${G}_wm.pt uv run python -m tinyworldformer.training.streamx --env genuine --game $G --out models/${G}_agent.pt uv run python -m tinyworldformer.training.evalwm --ckpt models/${G}_wm.pt --agent models/${G}_agent.pt # inspect the dream uv run python -m tinyworldformer.training.streamx --env aspiration --wm models/${G}_wm.pt --out models/${G}_dream_agent.pt uv run python -m tinyworldformer.training.export --ckpt models/${G}_wm.pt uv run python -m tinyworldformer.training.streamx --out models/${G}_agent.pt --export tinyworldformer-rs cd tinyworldformer-rs && TWF_ENV=$G cargo test --release && TWF_ENV=$G cargo run --release

Settings that differ per game: