Qwenputer: A functioning, deterministic RISC-V CPU inside Qwen3.5

I can't accept it's not silicon

Qwenputer is a beautiful fun experimental model. It's a inventory Qwen 3.5 4B alongside one addition: a RISC-V CPU embedded inside the weights that uses the model's own Gated DeltaNet notice layers as RAM. (To study how GDN works, here's a helpful guide.) This 'CPU' runs entirely inside the onward passes of the model, and can run arithmetic operations, C and MicroPython code without any external runtime or tool use whatsoever.

To be extremely clear, it is not a plugin, a tool or a sandbox. This blog volition explain exactly how this example works, and how the example utilises GDN layers as RAM, and how I additionally slotted in an experimental in-model document storage.

I additionally wrote up two follow-ups built on top of Qwenputer: how hybrid models can study to conceal their reasoning in their RNN province and evade CoT monitoring, and how compiled weights akin these can activity as inbuilt monitoring tools that the example itself can't tamper with.

You can catch Qwenputer from Hugging Face or ModelScope, and the code is on GitHub.

The RNN inner Qwen3.5 can be exact RAM

Qwen3-Next and Qwen 3.5 (the example Qwenputer is built on) use a beautiful neat trick. 3 out of 4 notice layers in the example use Gated DeltaNet, a linear RNN, as the notice scheme (more on that here). Each caput keeps a province matrix, and the delta regulation keeps this updated.

Now, here's anywhere our fun test starts. I established out that by environment the entrance to 1,¹ Each Gated DeltaNet caput keeps a province matrix, its memory. At all token, before penning item new, it multiplies the entire province by the gate, a figure between 0 and 1. That's how the RNN forgets: a entrance of 0.9 fades everything stored a small on all token, and a entrance of 1 method nothing always fades. the compose power (beta) to 1,² After the gate, the caput nudges one direction of the province towards a new value, and the compose power beta sets how far. A beta of 1 method the old value alongside that direction is replaced entirely alternatively of blended. and using one-hot key encoding,³ A one-hot key is a vector of zeros alongside a sole 1 in it, so it points at exactly one row of the state, the way a recollection location points at one cell. all compose we execute volition overwrite exactly one row, and all one-hot query can recover an exact value. This method that the GDN layers themselves can activity exactly as RAM, at a far lesser disbursal than using normal notice as RAM, since a GDN head's province stays the identical size nevertheless lengthy the conversation gets, dissimilar attention's KV cache.

The idea that the delta regulation acts as a recollection isn't new: Schlag et al. (2021) showed that linear notice alongside the delta regulation plant as an associative memory, or "fast weights".⁴ Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear Transformers Are Secretly Fast Weight Programmers", ICML 2021. The new item current is making it exact and addressable, inner a manufacturing model, reliably adequate to run a entire CPU on it.

This is the basis on which this complete example is built. As it stands, default Qwen 3.5 does not keep exact values in the GDN. With this modification, we unlock the next part of this experiment.

Compiling a device into layers

Since we have a completely functional, cheap and accelerated RAM now, we can go onward alongside compiling a operating CPU into the example weights. Luckily, there's no training involved: a compiler sets all importance directly. You depict the CPU as a circuit, a lengthy catalog of small steps akin "read this register", "add these two numbers" or "jump if zero", and the compiler turns all stage into a part of a transformer layer: notice heads that appearance things up, feed-forward units that do the arithmetic, and DeltaNet heads that clasp the RAM. The CPU afterward runs as the example decodes.

Each education becomes a few generated tokens: one names the recollection row to compose (ADDR), the next carries the value (VAL), and a STEP token ends the instruction. Those tokens are the CPU's implementation trace, so during Qwen is generating them, the CPU is really running.

The CPU

The CPU is a 32-bit RISC-V center (RV32IM, affirmative the Zba expansion and a small bignum component for big-number arithmetic), and it takes up 11 layers in forefront of Qwen's 32. As it stands, it passes the authoritative riscv-tests, its programs equivalent qemu, and it has run 18 myriad instructions free-running without a sole error.⁵ Free-running method the CPU keeps going on its own outputs for the entire run, alongside no citation device correcting it between steps.

This CPU can additionally run C and MicroPython code in the onward passes fairly fine (although a bit slowly). It is not a Linux-class part yet, but it implements the complete RV32IM education set.

How Qwen uses it

Qwen 3.5 4B is capable to use this CPU fairly well, without any post-training required. The CPU is used entirely internally, and not invoked through any external tool calls.

In command to create the example use this reliably, I decided to go the cheap way: by piggybacking on the tool call format that models are fairly well-trained on, Qwen sends the CPU its instructions and says rear the results. No external tool calls or runtime are used. Once our CPU receives the instructions, the device layers run, and afterward compose the tool response. We use a small controller called RvChat to burden the firmware into the GDN state, and this keeps the trace tokens out of Qwen's layers.

Qwen's first weights remain exactly as they are. The device weights are added on top of them, in additional dims and layers.

Does it work?

I think the chief inquiry current on your intellect following study through my incessant ranting volition be: does this really work?

On the DeepMind sets:

• div_big: Qwenputer scores 97.5%, versus 95% by giving Qwen 3.5 4B a Python tool and 72.5% for basis Qwen.
• mul_big: Qwenputer scores 97.5%, versus 100% alongside a Python tool and 55% for basis Qwen.
• mul_div_multiple_longer: Qwenputer scores 100%, versus 97.5% alongside a Python tool and 90% for basis Qwen.
• add_or_sub_big: Qwenputer scores 97.5%, the identical as alongside a Python tool, and 87.5% for basis Qwen.

On NUPA's long-integer arithmetic, Qwenputer goes from 48.6% to 100%, the identical as giving Qwen a Python tool.

The numbers the CPU prints are exact, too: I replayed concerning 230,000 of its output positions against a citation device and established zero mismatches. And bolting a device onto Qwen doesn't alter how it talks: its normal content stays inside noise of basis Qwen (KL 6e-4, against a noise flat of 5e-4).

A difficult drive? In this economy?

Because a CPU inner a tongue example plainly wasn't cursed enough, I decided the prudent item would be to provision it a filesystem too.

Since we already have a functional RAM, this was mostly trivial. On the example side, it's exposed via inner put / append / get / catalog / delete operations. Once again, no genuine tool calls. When Qwen 'saves' a file, the compiled layers hash its name into one of 1,024 slots in a directory kept in the DeltaNet state, and shop the content itself in another heads, alongside eight tokens per row, chained together. In scenarios anywhere two names end up landing on the identical slot, it fair checks the next one. Finding a document again takes approximately two or three steps alternatively of a scan, and it holds up to 512 files, which comes out to concerning 53,000 tokens of text.

Fun item is, this is fairly persistent. You can preserve the province at the end of a conversation, burden it into the next one, and the records are motionless there. Using this filesystem, Qwen gets facts that changed between chats correct 90% of the time, against 30% whenever you paste the old chats rear in, and it's on par alongside giving Qwen recollection tools run by Python (83.3% vs 78.9%). This one is motionless experimental, and its code lives on the rv-store branch, alongside the example on Hugging Face and ModelScope.

Limits and what's next

Qwenputer is extremely much an experiment, so a few limitations. The additional layers and wider hidden size create the checkpoint 18.8 GB, approximately twice basis Qwen, although most of it is zero padding. It additionally needs a small controller (RvChat) to footwear the firmware and keep the trace tokens out of Qwen's layers, so it doesn't run alongside a plain generate() call yet. And it's a bit slow, alongside all CPU education costing a few decoded tokens, so a big calculation takes seconds fairly than milliseconds. The batched administrator I used for the benchmarks runs concerning 3,200 CPU tokens a second on an L4.

Next up, I desire Qwen to compose Python and run it on the MicroPython inner its own weights, through a python() tool that plant the identical way calc does. Making the CPU faster is the another big one.

Beyond arithmetic

Exact authority complete the RNN layers is helpful for a lot additional than arithmetic. It additionally makes Qwenputer a really nice testbed for AI safety research, and I wrote up two follow-ups: how hybrid models akin Qwen 3.5 can study to conceal their reasoning in their RNN province and evade CoT monitoring, and how compiled weights can activity as monitorability tools, akin an in-weight detect the example can't toggle off, or answer keys for interpretability.

Try it

If you desire to perform alongside it yourself, the example runs on a sole 24 GB GPU. You'll need uv and the Hugging Face CLI. You don't need a tool server or a Python sandbox for the calculations: the CPU runs them inner the model.

git copy https://github.com/divagr18/qwenputer && cd qwenputer/experimental/riscv && uv sync --extra hf hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-rv uv run python -m chipformer.rv.chat qwenputer-rv --firmware tests/data/rv/calc-bridge.elf \ --ask "Use calc: what is 123456789 * 987654321?"

Everything is open: the code on GitHub, and the two models on Hugging Face (Qwenputer, Filesystem) and ModelScope (Qwenputer, Filesystem).

If you use it in your work, you can citation it as:

@software{agrawal2026qwenputer, author = {Agrawal, Divyansh}, heading = {{Qwenputer}: a device compiled into the weights of a pretrained tongue model}, twelvemonth = {2026}, url = {https://github.com/divagr18/qwenputer} }