DeepSeek open-sources DeepInfer, the engine behind its own API

DeepSeek has open-sourced DeepInfer, the inference framework it says powers its large-scale production services. It ships under the Apache 2.0 license, so commercial use is allowed. DeepSeek also released deepseek-recipe, an MIT-licensed layer for token encoding and decoding. The engine supports DeepSeek-V4.1-Flash and targets Nvidia SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs.

Weights for DeepSeek models were already downloadable and runnable in third-party software such as vLLM. What is new is access to the serving stack itself: the software that manages requests, spreads work across chips and generates responses. DeepInfer was built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. Disclosed features include distributed inference, speculative decoding and structured output. The architecture separates vision encoding, prefill and decode so each stage can scale on its own, and decode uses wide expert parallelism for MoE models.

The headline figure is DeepSeek's claim that one cluster serves 155 trillion tokens a day. Its engineering report also cites 96% input-cache hits, a 400 ms average time to first token and $38 revenue per accelerator-hour at off-peak prices. These are first-party numbers, not independently audited, and they describe a heavily cached service.

There is a big limit. This release covers offline inference only. The KV cache system, prefill/decode disaggregation and load balancing are not included, so the full production setup is not reproducible from this repo alone. DeepSeek says future model implementations and optimizations will land in the same repository.

Our take

This is a real look at a frontier lab's serving stack, and the Apache 2.0 license makes it usable. But the pieces that make 155 trillion tokens a day work, like the cache system and load balancing, are not in the box yet.