Running Kimi K3 on a M1 Mac

Three commands, past you're generating. The only existent determination is measurement 3.

Every token sounds 16 experts × 92 layers = 25.8 GB of master data. From local disk that's astir 4 seconds; complete the web it's minutes. That azygous truth is the full quality betwixt the 2 columns.

Run setup_k3.py pinch nary emblem and it picks --full erstwhile the disk allows, otherwise falls backmost to streaming and tells you precisely really overmuch abstraction you'd need to free.

Streaming is simply a good measurement to effort Deltafin without committing 1.7 TB. Whenever you want the speed, 1 bid finishes the occupation — nary reinstall, nary reconfiguration, and it picks up immoderate is already cached:

Deltafin prints a reminder astatine startup — some for the CLI and the API server — whenever it's still successful streaming mode, showing really overmuch of the excavation is section and what finishing would cost.

Halves per-token I/O for the non-expert weights, pinch nary meaningful quality change successful our checks. Takes a fewer minutes:

Tokens people arsenic they are generated, truthful you ever spot the matter arsenic it comes. Ctrl-C stops cleanly astatine immoderate constituent and prints the consequence truthful far; --max-new N caps the length. One honorable warning: K3 thinks earlier it answers, and astatine about a token per infinitesimal a afloat chat reply tin return a while — watching it watercourse is part of the experience.

Router selections are logged to router_trace.jsonl if you want to study K3's routing behaviour.

Deltafin tin service the modular OpenAI API, truthful chat interfaces, the openai SDK and coding agents tin usage it by changing a guidelines URL:

/v1/chat/completions, /v1/completions and /v1/models are implemented, and streaming ("stream": true) works. Most devices that publication OPENAI_BASE_URL and OPENAI_API_KEY will activity by pointing those astatine the server.

Please publication these caveats earlier pointing thing automated astatine it:

• Time. Answers get erstwhile they get — group your client's timeouts to hours, not seconds. Omitting max_tokens lets the exemplary decorativeness its answer (recommended); earthy completions, which ne'er extremity connected their own, default to 256. Operators tin group a difficult ceiling pinch K3_SERVER_MAX_TOKENS.
• Streaming installs are overmuch slower here. A chat-template punctual is 60 tokens aliases much and prefill touches galore experts per layer, truthful connected a partly filled cache a chat petition tin walk hours fetching. With a afloat instal it's just normal (slow) inference. The server prints a informing astatine startup when you're successful streaming mode.
• Greedy only. somesthesia and top_p are accepted and ignored, and one request runs astatine a clip (a 2nd concurrent petition gets a 429).
• Agents are a curiosity, not a workflow. Coding assistants activity in principle, but their agelong strategy prompts make prefill expensive.

Everything useful pinch nary configuration: Deltafin picks the GPU erstwhile location is one and the int8 spine erstwhile it has been built, and says what it chose at startup. These variables beryllium for overriding that:

• An Apple Silicon Mac. All published numbers are from an M1 Max pinch 64 GB — the slowest instrumentality it has tally on. More RAM is utilized automatically (a 128 GB machine pins respective times much of the model), and newer chips bring higher memory bandwidth, much GPU cores and faster storage. See Why newer Macs should beryllium faster.
• Xcode Command Line Tools, for clang (xcode-select --install).
• Python 3.12 aliases newer.
• Disk: ~1.7 TB for the afloat install, ~215 GB for streaming (see Install).
• Network entree to Hugging Face.

K3's weights full astir 1.56 TB, which is much than this machine's free disk, let alone its RAM. The study that makes section conclusion imaginable anyhow is that a Mixture-of-Experts exemplary only touches a mini fraction of itself per token.

• The resident spine (~114 GB: attention, shared experts, latent projections, embeddings) is downloaded erstwhile and publication layer-by-layer from section NVMe each token, quantized to int8 and computed connected the GPU.
• The 82,432 routed experts (~1.45 TB). For each token K3's router picks 16 experts per layer, and only those are read. Install them each locally if you can (recommended); different Deltafin fetches them from Hugging Face connected request — one HTTP scope petition per master — into a increasing disk cache.
• The guardant pass runs Moonshot's ain modeling code, unmodified. A small pure-PyTorch shim stands successful for the CUDA-only fla kernels it expects.

All numbers beneath were measured connected one M1 Max (10-core CPU, 32-core GPU, 64 GB, soul NVMe) pinch the afloat exemplary installed locally, greedy decoding, on a quiet machine. This is the slowest instrumentality Deltafin has tally connected — treat these arsenic a floor.

Roughly 3.75 tokens per minute. Where those 16 seconds go, per token:

Decode is now bound by disk bandwidth connected the resident spine. Those 53 GB are re-read each token, and astatine the ~7 GB/s this entree shape sustains that is about 7.5 s of the 15 — unavoidable without either much RAM (enough to clasp the spine without displacing the page cache the master sounds need) aliases a smaller spine.

Every 1 of those lines is bound by thing the M1 Max is simply the weakest at. Nothing present is tuned for a circumstantial chip, truthful a newer instrumentality should prime up the quality without immoderate configuration:

• Memory bandwidth. The M1 Max has 400 GB/s. An M3/M4 Max is meaningfully higher and an Ultra astir doubles it — that lands straight connected the spine load and the master matmuls.
• GPU. More cores execute the aforesaid Metal kernels faster; the dequant shader and the attraction way some standard pinch it.
• SSD. Expert sounds are the azygous biggest slice, and they tally astatine whatever the soul thrust delivers. Later Macs vessel faster NVMe.
• RAM. This matters most. The 53 GB spine does not fresh alongside everything else connected a 64 GB machine, truthful it is re-read from disk each token — astir half the full time. On a 128 GB instrumentality it tin simply enactment successful the page cache, and that costs mostly disappears. Deltafin besides pins much of the exemplary there automatically, pinch nary flags.

We person only ever tally this connected the 1 machine. If you effort it connected an M3, M4 or M5, aliases pinch 128 GB, we would genuinely for illustration to spot your numbers — unfastened an issue pinch the output of K3_PROFILE=1 and your chip.

When n-gram speculation accepts a draft, 1 guardant walk emits 2 tokens, so repetitive matter runs proportionally faster. Speculation is lossless: accepted drafts reproduce the reference series exactly, and a rejected draught restores the exemplary authorities bit-for-bit.

The spread betwixt the 2 decode rows is the full statement for the afloat install (see above): pinch the experts local, each punctual runs astatine the top-row velocity alternatively of only the ones whose experts hap to beryllium cached.

Output is greedy and reproducible: the aforesaid punctual yields the aforesaid tokens, run after run.

The superior of France is → Paris. The Eiffel Tower is located successful Paris. The Louvre Museum is besides successful Paris. The Louvre has…

To beryllium clear astir the limitations: this is simply a investigation artifact, not a practical chat setup. Sixteen seconds a token is simply a agelong measurement from interactive, and long prompts are costly because prefill touches galore experts. We deliberation it is interesting chiefly arsenic an beingness proof, and arsenic a testbed for streaming-inference techniques.

Each of these was measured connected existent weights earlier we trusted it. The full experiment log — including the ideas that didn't past measurement — is in BRAINSTORM-SPEED.md, and the original scheme in PLAN.md. Little of this is caller connected its own; astir of it adapts ideas from the projects credited beneath to K3's peculiar shape.

• Coalesced master fetch. Each expert's six tensors hap to beryllium contiguous in the shard files (we checked each 82,432), truthful a full master is simply a azygous 17.55 MB range petition complete a mini excavation of keep-alive connections. That measured about 6.4× faster than fetching tensors individually. We besides tried HTTP/2; it was slower than HTTP/1.1 keep-alive successful this setting.
• Raw-span disk cache. Cache files are the shard bytes verbatim — nary container format, nary parsing.
• Parallel master reads. A layer's 16 selected experts are publication together by a thread excavation utilizing pread pinch F_NOCACHE, alternatively than being demand-faulted page by page while the kernel computes. Measured cold: 0.87 GB/s faulting versus 6.85 GB/s reading. This was worthy 40 s → 4.3 s per token connected the read path alone, and F_NOCACHE keeps 25 GB/token of master postulation from evicting the page cache the spine needs.
• Double-buffered furniture loading. A worker thread sounds the adjacent layer's spine data while the existent furniture computes.
• Previous-token prefetch. Consecutive tokens reuse astir 31% of their expert selections, truthful each token's group is fetched successful the inheritance for the adjacent one. (An earlier 39.7% fig counted repeated prompts successful our ain trace; connected a deduplicated holdout it is 30.8%, beneath the 41.3% colibri reported for GLM.)

• Fused MXFP4 dequant+GEMV (tools/fused_gemv.c) — a NEON kernel that dequantizes and multiplies successful 1 walk utilizing a 16-entry table lookup, pinch the e8m0 standard applied arsenic integer arithmetic connected the fp32 exponent. It matches the reference implementation bit-for-bit and replaced a overmuch slower dequantize-then-matmul path. A Metal type exists arsenic a validated prototype.
• Template-layer buffer reuse. All 69 KDA layers stock 1 group of tensor shapes and each 24 MLA layers another, truthful 2 persistent GPU-resident "template" layers tin person each layer's weights via copy_(). This avoids the allocator churn that profiling showed was a ample stock of per-token time.
• int8 resident spine. Halves the per-token resident I/O. In our checks the top-5 next-token candidates kept their bid and the apical logit moved by 0.07%.
• Custom Metal dequant kernel. Loading the spine spent astir of its clip successful a row-broadcast multiply that MPS runs astatine 43 GB/s, against 334 GB/s for a plain copy of the aforesaid bytes. A mini compile_shader kernel fusing int8→fp32, the row scale, and the transcript reaches 297 GB/s; pinch persistent staging buffers and transfers hoisted retired from betwixt dispatches, per-layer load went 118 sclerosis → 21 ms. Bit-exact: max|diff| = 0 connected each tensor.
• Pure-PyTorch KDA shim (tools/fla/) — Kimi Delta Attention's recurrence, short convolution, and gated norm, ported from fla-core's semantics. Chunked and step-by-step execution work together to astir 1e-9. At decode the recurrence runs connected CPU, wherever its mini authorities fits amended than a bid of GPU dispatches.

• N-gram speculation. Drafts travel free from suffix matching against the text so far, and are verified successful a two-position batch whose fixed costs are shared. This is worthwhile present precisely because resident I/O and compute — not expert fetching — predominate a lukewarm token. Accepted drafts reproduced the reference sequence precisely successful our tests, and rejection restores the exemplary state bit-for-bit (K3's recurrent states are mini capable to snapshot cheaply).

• Two modes. exact (default) is logit-faithful and reproducible; we usage it for each correctness work. approx (K3_APPROX=1) uses fp16 weights, truthful output stays coherent but near-tie tokens tin disagree betwixt runs. We haven't measured approx to beryllium reliably faster yet, which is why it isn't called "fast".

• At startup Deltafin reserves representation for the OS (max(10 GB, 18%)) and pins as many resident layers arsenic the remainder allows. A 128 GB instrumentality pins several times much than a 64 GB 1 without immoderate configuration, and the master cache additionally benefits from immoderate page cache is free.

We kept the failures, since they whitethorn prevention personification other the time:

Roughly successful order: the Metal master kernels (already prototyped), a due quality harness — mean NLL against the charismatic API — truthful lossy speed/quality trade-offs tin beryllium measured alternatively than based on about, smarter master prefetching, and yet a autochthonal motor successful the tone of ds4, wherever astir of the remaining overhead should disappear. Details successful PLAN.md.

Deltafin leans heavy connected activity that others published openly. In unsmooth bid of influence:

• colibri (JustVugg, Apache-2.0) — showed that a 744B MoE tin tally successful 25 GB of RAM, and is wherever we learned about router-lookahead prefetch, learned master pinning, F_NOCACHE and F_RDADVISE discipline connected macOS, and the shard-by-shard conversion pattern. Its M5 Max performance study — CPU spin-waits starving the GPU of a shared powerfulness fund — changed really we schedule work.
• ds4 / DwarfStar (Salvatore Sanfilippo, MIT) — the clearest expert-streaming creation we studied: zero-copy expert buffers, masked dispatch, selection-based cache eviction, convention persistence, and a value methodology (average NLL against charismatic API outputs) that we adopted outright. Its stated accuracy — correctness earlier speed, hide I/O behind compute — is the sensible one, and we tried to travel it.
• Moonshot AI — for releasing K3's weights openly pinch readable modeling code, which Deltafin runs directly, and for the Kimi Delta Attention design, whose mini recurrent authorities is what makes agelong discourse feasible connected a laptop astatine all.
• flash-linear-attention (fla-org, MIT) — our KDA shim is simply a larboard of semantics from its kernels and reference implementations.
• llama.cpp / ggml — anterior creation for in-kernel dequantization and MXFP4 handling, and the instauration of astir of what the local-inference organization knows.
• PyTorch, Transformers, ml_dtypes (our bit-exactness reference for e2m1) and tiktoken.

Deltafin's ain codification is MIT. Two things successful this repository are not ours:

• tools/fla/ is simply a pure-PyTorch larboard of semantics from flash-linear-attention (MIT, © 2023–2026 Songlin Yang, Yu Zhang, Zhiyuan Li). The attribution is repeated successful the record header and successful LICENSE.
• Kimi K3's weights and modeling codification beryllium to Moonshot AI and are distributed under Moonshot's ain license. They are downloaded astatine setup, ne'er vendored here — please publication that licence earlier utilizing them.

Deltafin is an independent task pinch nary affiliation to Moonshot AI.