DeepInfer is the high-performance conclusion example that powers DeepSeek's large-scale manufacturing services. This open-source publish provides offline conclusion alongside the center features needed for serving at scale.
Supported models:
- DeepSeek-V4.1-Flash
Supported features:
- Optimized kernels, example and sampler runtime
- Parallelism: Attention DP / stable CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead lot scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured outputs
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and tradition TileLang kernels.
- DSpark and confidence-scheduled verification: accelerate decoding alongside outline tokens and use confidence estimates to agenda verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum outline dimension (default: 5).
- Decoder SWA bounded replay: decrease decoder prefill activity by replaying lone a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must shield the SWA opening affirmative the maximum backtracking length.
- Asynchronous scheduling: overlap CPU lot scheduling alongside equipment execution. Enabled by default.
- Batch-invariant inference: keep example results identical throughout distinct lot compositions. Enable alongside DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
- Deterministic sampling: create identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and continue a fixed seq_seed to SamplingParams (NVIDIA only).
- Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.
Supported hardware:
- NVIDIA SM100 GPUs
- Huawei Ascend 950 NPUs
Required for a complete conclusion assistance (not included in this release):
- A KV cache system
- VE/Prefill/Decode disaggregation (most required motor functionality is already implemented)
- Load balancing throughout data-parallel replicas for the two prefill and decode
- Linux
- GCC >= 13.0
- Python 3.12
- CUDA 13 (NVIDIA only)
- CANN 9.2 (Ascend only)
Before construction DeepInfer, delight create and activate an bare Python 3.12 environment.
# Use `ascend` alternatively for the Ascend platform # Keep it exported in the surroundings anywhere you run DeepInfer export DEEPINFER_ACCELERATOR=nvidia ./build.sh
The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the archetypal run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.
python examples/offline_inference.py
See examples for details.
Measured on 4 SM100 GPUs for prefill and 64 for decode — all inside one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.
See eval/README.md.
DeepInfer is released under the Apache License 2.0. See LICENSE for details.
DeepInfer was initially built chiefly on vLLM, alongside additional inspiration from SGLang and TensorRT-LLM. We appreciate their developers.
@misc{deepinfer2026, title = {DeepInfer: High-performance, large-scale conclusion example for DeepSeek models}, author = {DeepSeek-AI}, year = {2026}, publisher = {GitHub}, howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}}, }
