What Kimi K3 Costs to Run

Introduction

Kimi K3, released successful July 2026 by Moonshot AI, is the largest open-weights exemplary to date. As a Mixture of Experts (MoE) exemplary pinch 2.8 trillion full parameters and only ~104B progressive parameters, Kimi K3 has a mini ratio (3.7%) of active-to-total parameters compared to different frontier open-weights models. This architecture gives it a batch of imaginable intelligence without needing to activate arsenic galore parameters successful a azygous request.

The comparatively mini number of progressive weights does not mean it is easier to download and tally locally though. It still requires a monolithic magnitude of RAM to shop the full weights. This article discusses what it really takes to tally and erstwhile the mathematics could make consciousness to self-host connected rented GPUs vs. utilizing serverless inference.

The Parameter Count Is Not the Whole Story

Most models are trained successful 16-bit precision and acquisition a simplification successful intelligence if you tally them successful 4-bit precision to prevention memory. Kimi K3 was trained pinch autochthonal Microscaling Floating Point 4 (MXFP4) quantization-aware training (QAT) pinch 4-bit merchandise weights by design, truthful it tin beryllium tally much efficiently without lossy compression. In Kimi K3’s QAT, each guardant walk rounds the weights to 4-bit earlier utilizing them, truthful the nonaccomplishment that the exemplary is optimizing includes the quantization difference. Therefore, the exemplary learns to beryllium bully astatine 4-bit, alternatively than accepting a level of correction erstwhile moving successful 4-bit alternatively of 16-bit.

You tin tally Kimi K3 successful 8-bit aliases 16-bit precision arsenic well, but the logic to do truthful would not beryllium for improved quality. It would only beryllium for compatibility reasons. FP4 is for inference, truthful if you wanted to fine-tune aliases proceed training, you would besides request higher precision.

To get a basal estimate of the magnitude of RAM required to conscionable load a model, you tin multiply the full number of parameters by the number of bits of quantization you want to tally it at. Then disagreement that number by 8, because representation is measured successful bytes, and location are 8 bits successful 1 byte. In the lawsuit of Kimi K3, 2.8 trillion parameters x 4 bits divided by 8 is 1.4 terabytes of RAM for the weights alone.

If you would for illustration to tally it successful 8-bit aliases 16-bit precision, you will request double aliases quadruple this magnitude of RAM. With Kimi K3, you’d only do this to tally it connected hardware without autochthonal FP4 aliases to usage much mature package than the existent FP4 kernels. Kimi K2 variants, by comparison, were only 1 trillion parameters, but the afloat precision weights (BFloat16 (BF16)) required 2 TB of RAM, which is astir 600 GB much than K3’s native. Then further RAM is needed to shop the key-value (KV) cache, activations, and runtime overhead during inference.

The Architecture Is Built to Be Cheaper to Run Than Its Size Suggests

The logic a exemplary arsenic ample arsenic Kimi K3 is economical to service astatine each is because almost nary of it runs connected a fixed token. The ratio innovations make the costs mathematics work. The MoE sparsity allows only 16 of 896 experts (+ 2 shared) to activate per token, but it uses quantile balancing routing to dispersed tokens crossed experts deterministically, truthful nary master capacity is wasted. Kimi Delta Attention (KDA) keeps a fixed-size attraction authorities alternatively of per-token KV cache, cutting the KV-cache representation by up to ~75%.

Attention Residuals let each furniture successful the neural web to propulsion successful circumstantial outputs from earlier layers pinch learned weights. Stable LatentMoE is K3’s Mixture of Experts model wherever experts run successful a compressed latent abstraction to enactment cheap. KDA, Attention Residuals, and Stable LatentMoE activity together to summation the wide scaling ratio by ~2.5x complete Kimi K2.

Why This Isn’t a Single-GPU Problem

For a azygous user, moving Kimi K3 conclusion would require an further 2-15 GB for the KV cache, depending connected context, ~30 GB for activations, and ~30 GB for runtime overhead, totaling astir 1.5 TB of RAM including the 1.4 TB for the weights. A azygous GPU is retired of the question, leaving america pinch multi-GPU, and for realistic scaling, multi-node solutions for hosting.

Using basal arithmetic pinch H100’s 80 GB, and H200’s 141 GB, we would request 19 H100s aliases 11 H200s. However, moreover though Hopper (and MI300X) tin load 4-bit weights, they dequantize astatine runtime, keeping the representation savings but losing the velocity benefits that K3’s autochthonal 4-bit architecture allows. Using the newer generation, Blackwell B200/B300 aliases AMD MI350X/MI400s to execute MXFP4 (4-bit) natively is simply a amended approach. So self-hosting is simply a hardware procreation determination alternatively than conscionable a GPU-count decision.

vLLM recommends a minimum of 8x B300s aliases MI355X/MI350X (with Radeon Open Compute (ROCm)), multi-node for production. Also, because KDA’s unsocial prefix-caching process doesn’t person per-token KV caching, for illustration modular vLLM, mean prefix caching doesn’t activity seamlessly pinch it. The creators of Kimi K3, Moonshot, contributed a civilization vLLM implementation that solves this problem. In bid to self-host, you request a circumstantial caller vLLM build pinch the K3-specific code.

The Self-Host vs. API Decision Framework

Kimi K3’s first-party API is $3.00 input / $0.30 cached input / $15.00 output per 1M tokens, pinch DigitalOcean unrecorded astatine the aforesaid $3/$15. The cheapest native-FP4 azygous node is ~8x MI350X astatine ~$4.76/GPU-hour. This is astir $38/hr and $27,800/month connected a reserved GPU Droplet that is moving whether aliases not anyone sends a request. To meet this costs pinch Serverless Inference, a personification would request to usage astir 1.8B output tokens/month, aliases ~700 output tokens/second sustained astir the timepiece astatine the API’s $15/1M token rate. In reality, a single-stream tokens per 2nd (TPS) complaint is successful the tens, possibly hundreds of TPS successful perfect conditions, making it virtually intolerable to scope that constituent connected a azygous stream. However, connected a node, it is imaginable to service astatine slightest 40 requests successful parallel. With little than the afloat 1M token discourse per request, the full requests tin beryllium person to 600.

Heavy users mightiness person ~50M tokens/month costing ~$750 maximum via the API. This is connected the higher extremity considering the much costly output tokens only. For a azygous user, it is astir 40x cheaper to usage serverless conclusion done an API than self-hosting connected rented GPUs. However, erstwhile you walk 40+ consistent, dense users astatine 50M + tokens/month, the mathematics starts to alteration successful favour of self-hosting. These 40 users could beryllium individuals aliases they could correspond an elaborate supplier orchestration pinch 40+ agents directed by a azygous user.

If we tin do 40 concurrent requests connected a node, tin we usage less GPUs truthful we only request capable GPUs for 1 user? The short reply is no, for a fewer reasons. First, serving models of this size splits each furniture crossed GPUs (tensor parallelism), and the mathematics useful good connected two, four, aliases 8 GPUs, alternatively than six (the minimum required to clasp the weights + conclusion requirements for a azygous user). Therefore, 8 is the minimum requirement. Secondly, you mostly can’t rent 5-6 of these GPUs. They are rented successful packages of 1 aliases eight. Even pinch six, the TPS mathematics would not make it imaginable to usage capable tokens to warrant the costs alone.

For immoderate users, the costs are not the main priority. Instead, control, consistency, reliability, information residency, and different factors mightiness make self-hosting much attractive. In that case, it is worthy it to look into the Terms of Service for the conclusion supplier you’re using. Many third-party providers don’t shop conclusion information and person fantabulous information privateness records, truthful self-hosting connected rented GPUs mightiness not beryllium necessary. If you scheme to self-host Kimi K3 and connection it commercially arsenic a service, you should besides cheque the Kimi K3 license. It is simply a civilization Kimi K3 licence arsenic opposed to Apache aliases MIT, and it comes pinch commercialized gates supra definite gross ceilings.

What This Means If You’re Running Kimi K3 connected DigitalOcean

If you take to spell pinch serverless inference, DigitalOcean offers Kimi K3 connected the Serverless Inference API arsenic a pay-per-token action astatine the $3/$15 rate. If you take to spell the self-hosted GPU route, you tin usage DigitalOcean’s GPU Droplets, Dedicated Inference, aliases semipermanent reserved GPUs. With GPU Droplets, you negociate the stack, and pinch Dedicated Inference, DigitalOcean runs a managed vLLM endpoint that you bring the Kimi K3 weights to. The 4-bit autochthonal options DigitalOcean provides are the AMD Instinct MI350X and the NVIDIA B300. Availability tin alteration without notice, truthful beryllium judge to cheque what is presently disposable and connected which regions earlier readying your deployment.

If the GPU Droplets and Dedicated Inference options are astatine capacity, please scope retired to the DigitalOcean sales team astir dedicated capacity. DigitalOcean besides offers Bare Metal GPUs, but does not presently connection FP4 autochthonal options successful the bare metallic format, truthful they are not the correct fit.

Conclusion

With larger unfastened weights models reaching the capabilities of closed frontier models, existent infrastructure decisions that utilized to beryllium invisible down an API are now becoming portion of the strategy for galore teams. With a capable number of users and precocious capable tokens per month, self-hosting tin supply costs savings, control, and thief alleviate privateness concerns. Even aft Kimi K3 is nary longer a apical contender for your team, the self-hosting vs. Serverless Inference determination is 1 astir teams request to make arsenic open-weights models go progressively capable. The aforesaid comparative logic covered successful this article applies to these early models arsenic well.

• Kimi K3 Model Card (Hugging Face)
• A Preview of Production-Scale Kimi K3 Support connected vLLM
• DigitalOcean GPU Droplets