DeepSeek Open-Sources DeepInfer, the Inference Engine Behind Its Services

Chinese AI lab DeepSeek has open-sourced DeepInfer, the software it uses to run its own models at scale. The code went up on GitHub on 20 October 2026 under the Apache 2.0 licence. According to its README, “DeepInfer is the high-performance inference framework that powers DeepSeek’s large-scale production services.”

The release is narrower than that description might suggest. DeepSeek says it “provides offline inference with the core features needed for serving at scale”, but several pieces needed for a full serving system are left out. It currently supports a single model, DeepSeek-V4.1-Flash, on two kinds of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

What DeepInfer includes

Inference is the work of running a trained model to produce answers. The README lists the features in this release:

• Optimised kernels, model and sampler runtime
• Parallelism: attention DP / balanced CP, MoE EP + EPLB
• DSpark and confidence-scheduled verification
• SWA bounded replay
• Asynchronous scheduling (a “zero-overhead batch scheduler”)
• Batch-invariant inference and deterministic sampling
• Structured outputs

The optimised kernels draw on DeepSeek’s own open-source libraries, including FlashMLA, DeepGEMM and DeepEP, plus TileKernels, DeepSelect and custom TileLang kernels. DSpark and confidence-scheduled verification “accelerate decoding with draft tokens and use confidence estimates to schedule verification”, with a default maximum draft length of five.

Two features target reproducibility. Batch-invariant inference is meant to “keep model results identical across different batch compositions”, while deterministic sampling aims to “produce identical outputs for identical input tokens and sampling parameters”. The README marks both as “NVIDIA only”. Structured outputs can constrain generation to a regex, JSON schema or EBNF grammar.

What is missing

DeepSeek is explicit about the gaps. Under the heading “Required for a complete inference service (not included in this release)”, the README lists:

• “A KV cache system”
• “VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)”
• “Load balancing across data-parallel replicas for both prefill and decode”

These are significant omissions for anyone hoping to run a production service. The KV cache stores intermediate results so they do not have to be recomputed, and disaggregation splits the compute-heavy prefill stage from the decode stage across different machines. Teams wanting a full stack would need to build or bolt on those pieces themselves.

Hardware, set-up and benchmarks

The supported hardware is exactly two platforms: “NVIDIA SM100 GPUs” and “Huawei Ascend 950 NPUs”. Prerequisites are Linux, GCC 13.0 or later, Python 3.12, and either CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The default checkpoint is DeepSeek-V4.1-Flash-1017, which is downloaded from Hugging Face on first run.

The README supports Huawei’s Ascend chips alongside NVIDIA’s, but does not compare performance between the two platforms; its published benchmark set-up uses SM100 GPUs.

DeepSeek publishes its performance benchmarks only as charts, with prefill and decode results “Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain.” It gives no headline speed figures, and provides a script for others to reproduce the tests.

The project also acknowledges its roots. “DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM,” the README says. Those are widely used open-source serving engines.

Our read

Publishing the engine that runs its own services fits DeepSeek’s pattern of open-sourcing infrastructure, not just model weights. For researchers, the reproducibility features and Ascend support are the most interesting parts. But with one model, two hardware targets and no KV cache, disaggregation or load balancing, this is a strong reference implementation rather than a drop-in replacement for vLLM or SGLang in production.

Sources

1. GitHub: deepseek-ai/DeepInfer, “DeepInfer: High-performance, large-scale inference framework for DeepSeek models” (README, 20 October 2026).

Related stories

• DeepSeek Releases DeepSeek-V4.1-Flash-1017 Open-Weight Update
• DHS Orders DeepSeek Out of US Civilian Agency IT Systems