DEDUPED Hot AI repos, turned into use cases Get the Monday 5
19,304 stars · Python

How to use Deep Infer: what you can do with it

DeepInfer is an open-source, high-performance offline inference framework from DeepSeek for running DeepSeek models, currently DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

#1 in this week's top 10 trending AI repos

View repo on GitHub

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer

What do you want to do?

4 use cases

  • Run offline inference on DeepSeek-V4.1-Flash locally

    Build Deep Infer from source and generate text from the default DeepSeek checkpoint.

    View prompt
    Clone https://github.com/deepseek-ai/DeepInfer, set up a Python 3.12 environment, export DEEPINFER_ACCELERATOR=nvidia, run ./build.sh, then run python examples/offline_inference.py using the checkpoint at <checkpoint path>. Report the generated text and the build result, and verify that the script finishes without an import or device error.

  • Get repeatable model outputs for regression tests

    Use batch-invariant inference and a fixed seed so identical inputs give identical outputs.

    View prompt
    In the DeepInfer repo, set DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, pass a fixed seq_seed to SamplingParams, and run <prompt file> twice. Output a diff of the two result files. Verify that the diff is empty.

  • Force model output to match a JSON schema

    Use structured outputs to limit generation to a regex, JSON schema, or EBNF grammar.

    View prompt
    In the DeepInfer repo, follow examples/_structured_output.py to constrain generation with so_args for the schema in <schema file>. Output the generated JSON for the prompt <prompt>. Verify that the output parses with json.loads and matches the schema's required keys.

  • Reproduce Deep Infer prefill and decode benchmarks

    Run the sweep script to measure prefill and decode performance on SM100 GPUs.

    View prompt
    In the DeepInfer repo, run examples/sweep/run_sweep.sh on <number of GPUs> SM100 GPUs and save the output to <output path>. Report the prefill and decode results. Verify that the GPU count matches the layout the README describes before comparing with the published charts.

Quick start with Deep Infer

  1. Check you have what it needs: Linux with GCC 13.0 or newer and Python 3.12; NVIDIA SM100 GPU with CUDA 13, or Huawei Ascend 950 NPU with CANN 9.2; Access to the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint on Hugging Face, or a local copy.
  2. Open deepseek-ai/DeepInfer on GitHub and read the README and license.
  3. Create and activate an empty Python 3.12 environment.
  4. Run export DEEPINFER_ACCELERATOR=nvidia (use ascend instead on the Ascend platform), and keep it exported in the shell where you run Deep Infer.
  5. Run ./build.sh from the repo root.
  6. (Optional) Run DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to use a local checkpoint instead of the Hugging Face cache.

These steps come from our top use case, Run offline inference on DeepSeek-V4.1-Flash locally, which also has a copy-paste agent prompt.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for deepseek-ai/DeepInfer

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Deep Infer alternatives · Best AI repos for Software and DevOps

About this repo

  • Who it is forWritten for ML engineers and infrastructure teams with SM100 GPU or Ascend 950 NPU access who want to run or benchmark DeepSeek models locally.
  • Why it is hot19k stars, launched 6 days ago, talked about in 5 AI sources we track
  • LanguagePython
  • Sourcegithub.com/deepseek-ai/DeepInfer · Back to Hot repos

Requirements

  • Linux with GCC 13.0 or newer and Python 3.12
  • NVIDIA SM100 GPU with CUDA 13, or Huawei Ascend 950 NPU with CANN 9.2
  • Access to the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint on Hugging Face, or a local copy
  • Enough disk space and memory for the checkpoint and the build

Add this badge to your README

Maintain Deep Infer? Show readers where to find these use cases. Use cases on Deduped

Markdown 

[![Use cases on Deduped](https://dedupedai.com/badge/deepseek-ai/DeepInfer.svg)](https://dedupedai.com/use-cases/deepseek-ai-deepinfer/?utm_source=github&utm_medium=badge&utm_campaign=deepseek-ai-DeepInfer)

HTML 

<a href="https://dedupedai.com/use-cases/deepseek-ai-deepinfer/?utm_source=github&amp;utm_medium=badge&amp;utm_campaign=deepseek-ai-DeepInfer"><img src="https://dedupedai.com/badge/deepseek-ai/DeepInfer.svg" alt="Use cases on Deduped" height="20"></a>

A static image, no tracking script. About the badge

Questions about Deep Infer

What is Deep Infer?

DeepInfer is an open-source, high-performance offline inference framework from DeepSeek for running DeepSeek models, currently DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. It is an open-source project on GitHub: https://github.com/deepseek-ai/DeepInfer.

Who is Deep Infer for?

Written for ML engineers and infrastructure teams with SM100 GPU or Ascend 950 NPU access who want to run or benchmark DeepSeek models locally.

What can I do with Deep Infer?

4 use cases so far, for example: Run offline inference on DeepSeek-V4.1-Flash locally; Get repeatable model outputs for regression tests; Force model output to match a JSON schema. Each has steps and a copy-paste agent prompt.

What do I need to run Deep Infer?

Linux with GCC 13.0 or newer and Python 3.12; NVIDIA SM100 GPU with CUDA 13, or Huawei Ascend 950 NPU with CANN 9.2; Access to the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint on Hugging Face, or a local copy; Enough disk space and memory for the checkpoint and the build.

What are alternatives to Deep Infer?

Other hot AI repos in Software and DevOps: Harness2, Orca, Opencode, Graphify, Replica Skill.

Is Deep Infer safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.