Contents

DeepSeek’s DeepInfer is an open-source inference framework for DeepSeek-V4.1-Flash. Its hardware and supporting service requirements mean you should check whether it fits your environment and use case before getting started.

Key Points#

  • The supported model is DeepSeek-V4.1-Flash, with support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
  • Linux, GCC 13 or later, and Python 3.12 are required, along with CUDA 13 for NVIDIA or CANN 9.2 for Ascend.
  • Features include speculative decoding, structured output, and inference that is less affected by batch composition.
  • The public release does not include everything needed for a complete inference service, such as KV caching and load balancing.

What DeepInfer Can Do#

DeepInfer is a public release for offline inference, based on the framework behind DeepSeek’s large-scale production inference service. It provides kernels, model execution, samplers, and parallel execution mechanisms.

Supported features include attention DP and balanced CP, as well as MoE EP and EPLB. It also uses optimized kernels such as FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect. Currently, the supported model is DeepSeek-V4.1-Flash, so check compatibility first if you intend to run another model.

Preparing Your Environment#

The runtime requires Linux, GCC 13.0 or later, and Python 3.12. NVIDIA environments require CUDA 13, while Ascend environments require CANN 9.2. Create and activate a clean Python 3.12 environment before building.

For NVIDIA, specify the accelerator through an environment variable before running the build.

setup.sh
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

For Ascend, replace nvidia with ascend. The DeepSeek-V4.1-Flash-1017 checkpoint is downloaded to the Hugging Face cache on the first run. To use a local model, set DEEPINFER_HF_MODEL_DIR to its location before running the example.

run.sh
export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py

Settings to Speed Up Inference#

DSpark and confidence-scheduled verification accelerate decoding using draft tokens. Enable them with DEEPINFER_ENABLE_MTP_VARLEN=1 and adjust the maximum draft length with DEEPINFER_USE_NEXTN_DECODE. The default is 5. Asynchronous scheduling is enabled by default, overlapping CPU-side batch processing with device execution.

To reduce the decoder’s prefill workload, configure SWA bounded replay. For the default V4.1 configuration, set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The number of trailing tokens retained must cover the SWA window and maximum backtrack length, so verify these settings for your use case.

Reproducibility and Structured Output#

To keep model results consistent across different batch compositions, enable DEEPINFER_BATCH_INVARIANT=1 in an NVIDIA environment. To obtain identical outputs for the same input tokens and sampling settings, also set DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams. Both features are supported only on NVIDIA.

Structured output can constrain generated results to a JSON Schema, regular expression, or EBNF grammar. Pass so_args to SamplingParams, specifying schema_type and schema. See the examples in the repository for schema formats and sample code.

Scope of Service Deployment#

Benchmarks were measured using 4 SM100 GPUs for prefill and 64 for decode, all within a single scale-up domain. Other configurations may not deliver the same performance. To reproduce the benchmarks, refer to run_sweep.sh; see eval/README.md for evaluation methods.

The public release alone does not provide a complete inference service. You must separately provide a KV cache system, prefill/decode disaggregation, and load balancing across replicas. A practical approach is to first verify functionality and performance through offline inference, then identify the components needed to deploy a service.

Next Step#

Check whether your intended GPU or NPU is supported, build in a clean Python 3.12 environment, and try running python examples/offline_inference.py.

Source: https://github.com/deepseek-ai/DeepInfer

Contents