Or rather, I vibe-coded an FPGA into operating an LLM. And it… fair worked.
Friday. November 20, 2026
One of the additional entertaining things concerning programming lately is that coding agents let me activity in entirely new domains by giving me admission to new programming languages and paradigms. Lately, that has meant hacking on hardware, now that penning low-level application is much easier. So, in this blog post, I’m going to try to recount the cognition of vibe coding an FPGA, going through a sequence of “oh shit” moments multiple times, and giving myself a new appreciation for what’s imaginable if you have a ample token prosperity (thanks YC) and engaging problems to item the models at.
I overheard several founders lately conversation concerning how FPGAs work. I’ll acknowledge I was comparatively naive concerning them. I’d never really experimented alongside them in my startup or in college, so I knew small concerning them before. The essential idea I was acquainted with: you can program a circuit into a reprogrammable chip. So fairly than having a broad intent computing architecture (like a CPU) you can have a reprogrammable one, ie the configuration loaded onto the part determines how its logic and routing are connected, and that (in theory) enables achievement advantages. I additionally cognize that it took a mountain of C++ to compose and engaged manually analyzing waveforms for simulation and verification. I had never done this before although and it seemed cumbersome.
That’s concerning what I knew going into this.
Naturally, my ML brain idea “could we use these for AI inference”. I’m not the archetypal to think this — there are a few repos attempting to harbor ML models to FPGAs on Github. But I hadn’t really seen a writeup on whether coding agents are fine at penning the low flat C++ and Verilog code to program them and verifying the outputs so that they really run accurately. (Spoiler alert, they are amazing at programming them.)
So I decided I needed to get my hands on one. AWS has F2 instances alongside FPGA hardware, and that seemed easier than going and purchasing bodily hardware, so I asked Codex to rotation up a booking and got cooking. It had a comparatively constricted recollection of 16 GB of FPGA HBM. Around the identical time, I was operating alongside the DeepGrove founders who were discussing their new ternary importance architecture that is fine suited for recollection constricted cases. It’s SOTA on Mac hardware alongside genuine agentic tool calling, so I figured why not, let’s see if I can get this operating on an FPGA and accelerated adequate to be useful.
What followed was a haze delegate operating in Conductor for multiple days alongside me mostly fair encouraging it periodically during it the two (1) got a example completely operating and (2) risen its speed by 33× from the archetypal full-model bit-exact baseline to get it to a real-time applicable speed. It was among the most intellect blowing autoresearch loops I’ve run, ultimately penning approximately 3M lines throughout the test former and parsing 30M lines of output files, burning lone $10,000 in Claude and OpenAI tokens to run hundreds of experiments. Most of the code is C++ and several Verilog. It’s been likely a decade since the final period I wrote any grave amount of C++ and I’ve never written Verilog. I honestly don’t cognize the archetypal item concerning Verilog. Most of my prompts were not particularly clever: they consisted mostly of me saying things akin “you’re a superintelligent model, period to be additional creative” and “let’s go for 2x faster, you got this”. Sometimes I brought several ML knowhow akin “hey it’s period to regard speculative decoding” or “why is that operating on CPU, you should harbor it”. But for the most part, it was fair vibes and I’m beautiful certain that alongside adequate period the example would have figured out those things. Also, it seemed that the two Astra and Opus (new Fable wasn’t out yet) were beautiful capable at this and I could use them interchangeably. I’m certain person has a fine benchmark for which is smarter but the two passed my smart-enough threshold.
From “it works” to 213 tokens per second
Measured benchmark tokens/s · choose a stage to see what changed
Step 14 · 33.13× baseline
A caput loader that says the two of its recollection banks at formerly (+17.3%, measured on and off on one image), a runtime expert-prefetch policy, and the router-job timing fix; run one clock stage lesser (225/425 MHz) following a soak test established random faults at 231.25.
The diagram follows the measured optimization sequence: 6 → 213 tokens/s, or 33× the baseline.
How much did the individual direction help?
At one point, about whenever it got to 10 TPS, I messaged the DeepGrove founders concerning this, and they replied amazed that it was operating so easily, saying they had tried this a few months rear and the models were not fairly fine adequate to be particularly effective. They sent me a immediate that summarizes several of their lessons and I copy-pasted it into my meeting to see if it’d matter. It was basically the most important part of genuine direction I added to this session. The immediate is reproduced below.
Read the complete FPGA scheme prompthere is what person alternatively did for a compact bs=1 decode optimized fpga implementation for a compact qwen manner example alongside ternary weights. you should obtain this alongside a kernel of sodium chloride and choice things that you accept are acutally helpful to add to your plan. Please additionally dont do things if you dont think is helpful that obtain a lot of times akin runnning really lengthy test of a sequence exact simulation etc or akin bodily scheme since those can obtain akin hours. current are several ideas delight choice and choose use this as environment and not somehting that you shoudl be strictly guided by somethings that could help. prioritiize speed of akin getting this up and operating though..
MATMUL (TERNARY ENGINE)
- No multipliers. A 2-bit importance drives a 3:1 mux that picks x, ~x, or 0. The unused 4th code raises an error flag, so corrupt weights get caught.
- Negation is ~x + 1. Lanes emblem negatives, and the row reducer adds the +1s once per row alternatively of formerly per lane.
- Activations are grouped integers: INT12, or INT16 anywhere we needed more range (MLP downward projection). Reduce as integers, use FP32 collection scales, accumulate in FP32.
- The pre-projection RMSNorm is deferred: quantize the activation first, apply the normalization measure following accumulation. It’s cheaper and numerically equivalent.
- 8 cores in two clusters. Projections that are autonomous (gate/up) run at the identical period on distinct center groups.
- Result: concerning fractional the DSPs of an INT8/INT16 scheme at the identical rate, traded for additional LUTs.
MEMORY
- Weights are packed at 2 bits, 8x additional per transfer than 16-bit. At lot 1, weight fetch expenses far additional energy and period than the math.
- Weights stream from HBM direct into the compute lanes and skip the on-chip buffers (they’re used formerly per token). Activations are cached next to all center and reused throughout rows.
- HBM routing is one sequence per recollection conduit alternatively of a big crossbar. Lay out data so all 32 channels remain busy.
- On-chip buffers are typed slots alongside an definitive lifecycle (open -> compose -> commit -> peruse -> release). Each slot has a flank memory for scales/exponents. The buffer checks format, range and producer, which catches scheduling bugs in hardware.
- The KV cache is MXINT8: 8-bit values alongside one shared exponent per 32-channel block. That’s small adequate to stream, alongside accuracy held.
- For MoE: stripe all expert throughout all channel, and map expert ID to address at runtime alternatively of using a fixed program.
ATTENTION
- Attention streams complete the cache in 64-token tiles alongside an online softmax (FP32 operating max/sum). The complete mark matrix is never stored.
- Probabilities are UINT16 and the environment accumulates in FP32.
- Each notice component interleaves two autonomous jobs on one datapath (C-slow), which hides pipeline latency without duplicating hardware.
- Q/K caput norm and RoPE got their own small vector unit, so they don’t block the broad vector path. The V projection overlaps alongside that work.
- New K/V is appended to HBM as shortly as it’s produced. We used a sliding window to border cache reads.
SCHEDULING
- One controller issues activity in program order, alongside tags and barriers. Engines overlap lone whenever the buffer requirements are explicit, so we got overlap without races.
- For MoE: the router is the new crucial path, since nothing can fetch until it finishes. Keep it in FP32 (lower precision changed which experts were picked), and overlap any you can about it.
CORRECTNESS (maybe not neecessary to folow excellent close)
- We fixed a numerics agreement archetypal (formats, rounding points, operation order) and built hardware to equivalent it. Vector ops must equivalent the reference model’s command and precision, not fair be “close”.
- Verification ladder:
- Per-block cocotb component tests.
- One complete tier co-simulated against the genuine checkpoint.
- Multi-token, complete 24-layer decode compared token-for-token alongside a reference oracle.
- Byte-level accounting of KV cache writes.
- We kept simulation results, timing signoff and commission results separate, and never claimed one from another.
- Every published figure links to the study it came from.
PHYSICAL DESIGN (maybe not necessary for u)
- The part has 3 regions (SLRs). Keep broad datapaths inner one area and register all crossing. Long links use credits, not combinational ready.
- We placed engines next to their data: recollection logic near HBM, notice in the DSP-dense region.
- Screen logic degree (10 levels max at 250 MHz) correct following synthesis, and fix it alongside pipelining, sharding, or C-slow. Full place-and-route is saved for integration.
- Dev loop: small alter -> lint -> component test -> quick synthesis -> depth/timing inspect -> restructure.
The coding delegate diligently considered these asks, but it didn’t appear to create all that much of a difference qualitatively. Indeed, whenever I asked it afterward to peruse all of the logs and decide which of these instructions it really used, it said:
Read the agent’s retrospective: what it adopted and rejectedIts biggest scheme ideas were already in my scheme before you pasted it. You pasted it on Oct 29 at 21:50 UTC: 4,741 characters describing what person alternatively did for a dense, batch-1, ternary FPGA decoder. Within 70 seconds I had sorted it into ten ideas to clasp and five to reject, and written that into all scheme agent’s brief. The shorter comment from your companion Edward, concerning using lookup tables as logic gates, came 24 minutes before and remaining nothing on the speed line.
Design ideas I adopted: mostly confirmation.
-
Already planned: streaming weights direct into the compute lanes, operating rows in parallel, and study the notice cache in bursts were all in the scheme concise I’d launched 35–40 minutes earlier.
- Match what was afterward built:
- interleaved lanes in the router (step 3);
- a distinct component for caput norms and RoPE (step 4);
- a program-order sequencer (step 5);
- a registered border on all crossing between dies (steps 8 and 12).
- I can’t display these wouldn’t have happened without the notes.
Ideas I rejected: all five numerics shortcuts. Lower-precision activations, a deferred norm, an 8-bit KV cache, online softmax and 16-bit probabilities all alter the output bits. That is anywhere their scheme got its efficiency, and your bit-exact regulation ruled it out.
Process advice: this changed things correct away.
-
What you asked: not to do things that obtain hours.
-
What I stopped: a simulation that had run for concerning 24 hours, and two 250 MHz builds.
-
What I switched to: a abbreviated iteration of C tests, synthesis and a logic-depth check, alongside one place-and-route per integration.
-
Likely gain: it got hardware results sooner.
-
Possible cost, which I can’t prove: two hangs that lone RTL simulation could display reached the commission the next day, and the big integration misplaced 38 hours to them.
One genuine discover it triggered. While checking their recollection ideas, I established AWS’s own citation recollection design, which reaches 426 GB/s throughout all 32 recollection channels. The recollection subsystem was built from it.
In another words, for most of the genuine scheme ideas it is unclear but there is fine logic to accept the example would have figured them out, particularly since it came up alongside them many cycles afterward following multiple compaction cycles. But the procedure direction was helpful immediately for getting to hardware results sooner and expanding repeat speed. In another words, the humans are motionless helpful at defining how to iterate accelerated and what correctness looks like, and the agents are awesome to oversee the exploration.
The optimization log
Here are the changes the autoresearch iteration found. Interestingly, it’s mostly a sequence of humble improvements that compound 1.5x ish all time, fairly than one or two elephantine ones.
Baseline: the archetypal full-model bit-exact FPGA depiction (R8), served alongside host-side prefix caching.
1.0× from baseline
Host application only: per-command overhead cut from ~10 ms to 0.9 ms (persistent enroll mapping, streaming reads, no per-command SDK calls or fsyncs).
1.25× this stage · 1.25× from baseline
A 128-lane notice motor that says the KV cache in bursts alternatively of a term at a time, keeping the reference's exact addition order.
1.74× this stage · 2.18× from baseline
A parallel MoE router alongside an exact GCC-sort tie-breaker affirmative a streaming output caput alongside built-in argmax; moderate-effort routing let it run at 250 MHz.
1.46× this stage · 3.19× from baseline
A glue motor that parallelizes norms, RoPE, quantization and copies between engines in certified exact orders.
1.47× this stage · 4.70× from baseline
The complete new pipeline (streaming ternary projection lanes + a new sequencer replacing R8's authority path), following fixing two hangs that lone appeared in hardware.
1.63× this stage · 7.65× from baseline
Exact speculative decoding: the presenter drafts tokens by immediate lookup and the FPGA verifies multiple in one pass.
1.04× this stage · 7.97× from baseline
Fixed a relay that moved router weights between dies at 1/6 speed: router stall per tier 43,862 → 12,292 cycles.
1.14× this stage · 9.08× from baseline
Head shade v2 (start the next token on a provisional guess; capture misses early) affirmative cross-die timing fixes, enabling 237.5 MHz kernel / 450 MHz memory.
1.62× this stage · 14.68× from baseline
Exact speculative decoding alongside a genuine outline model: a cut-down Maple on the F2's presenter CPUs drafts ahead, and the FPGA verifies all token.
1.12× this stage · 16.42× from baseline
Doubled the recollection ports from 4 to 8 (two projection lanes, output caput striped throughout banks): 1.39× per clock, but congestion cut the clock to 181/375 MHz.
1.03× this stage · 16.91× from baseline
Hid the notice engine's KV fetches rearward its compute, and replaced the router's 2,048-step serial sum alongside an exact order-independent one (12,292 → 2,338 cycles per layer).
1.24× this stage · 20.98× from baseline
Lossless 5-trit importance packing (−20% importance bytes), shorter serial steps, and moving the output caput to the far die, which relieved congestion and allowed 225/400 MHz.
1.27× this stage · 26.63× from baseline
Re-speculation following head-shadow misses, speculative MoE expert prefill during the router, and a die rebalance (without the unused DDR controller) that ran 231.25/450 MHz on the two boots.
1.08× this stage · 28.63× from baseline
A caput loader that says the two of its recollection banks at formerly (+17.3%, measured on and off on one image), a runtime expert-prefetch policy, and the router-job timing fix; run one clock stage lesser (225/425 MHz) following a soak test established random faults at 231.25.
1.16× this stage · 33.13× from baseline
How it compares
It’s difficult to do a 1:1 difference throughout these configurations. I didn’t run the identical autoresearch iteration on all alternative, and I assume there is additional achievement to extract if I did. The item is to display how dilatory the FPGA was in its archetypal implementation, and how far the optimization iteration took it.
The measurements below are from November 15 on AWS, replaying the identical 8-chat × 3-turn benchmark. Prices are the on-demand rates recorded alongside that experiment. “Exact turns” counts outputs that matched the pinned CPU reference. The GPU implementations are included for environment but did not fulfill the identical bit-exact criterion but are beautiful near in practice.
What happens as the environment grows?
This is one generation starting alongside a 16-token immediate and continuing until the 8,192-token environment is full: 8,176 generated tokens. The five center columns display decode speed in tokens/s at the indicated environment dimension for multiple generations of iteration. The final pillar is wall-clock period for the entire run/
* For r8 and r9, the archetypal two columns arrive from separate, shorter runs generating 2,032 tokens. The another columns arrive from the full-context run.
r9 improves short-context speed, during the complete long-context run is approximately tied alongside r8.
What the example really says
These examples arrive from the final r9 image.
Arithmetic, alongside the steps
246 tokens · 229.7 tokens/s · 1.16 s end to end
Read the arithmetic transcriptUSER: What is 37 × 48? Explain the steps briefly. MODEL: <think> We need to compute 37 × 48. The person asks to explain steps briefly. So I'll do multiplication: 37 × 48 = 37 × (50 - 2) = 37×50 - 37×2 = 1850 - 74 = 1776. Or directly: 37×40=1480, 37×8=296, sum=1776. I'll current a concise explanation. </think> To multiply 37 by 48, you can interrupt it down: - 37 × 48 = 37 × (50 – 2) = (37 × 50) – (37 × 2) = 1850 – 74 = 1776. Alternatively, 37 × 40 = 1480 and 37 × 8 = 296, so 1480 + 296 = 1776. The final answer is **1776**.
A small Python
140 tokens · 220.0 tokens/s · 0.76 s end to end
System prompt: “You are a helpful assistant. Think briefly before you answer, afterward answer plainly and concisely.”
Read the Python transcriptUSER: Write a Python function is_palindrome(s) that ignores case, spaces and punctuation. MODEL: <think> We need to compose a Python function is_palindrome(s) that ignores case, spaces, and punctuation. So we need to display out non-alphanumeric characters, change to lowercase, afterward inspect if the resulting cord is equal to its reverse. Provide the code. </think> ```python def is_palindrome(s: str) -> bool: """ Returns True if the cord s is a palindrome, ignoring case, spaces, and punctuation. """ # Filter to keep lone alphanumeric characters and change to lowercase filtered = ''.join(ch.lower() for ch in s if ch.isalnum()) # Compare alongside its reverse come back filtered == filtered[::-1] ```
Where this puts us
This was a fun project. I’m not certain I’m always going to use this particular part architecture in this particular way, and I’m certainly not claiming that an 8,192-token-context example operating on an FPGA is the way I’d propose anyone use item akin this in production. But it’s beautiful epic that this just works. Capability akin this entirely shifts the overton opening for what a committed squad of engineers can activity on. It should average that entirely new types of ideas (scientific discoveries, products, startups) are imaginable if you can item the models in the correct direction. I’m enthusiastic to see what everyone builds.