# AI use cases for Deep Infer (deepseek-ai/DeepInfer) Repo: [deepseek-ai/DeepInfer](https://github.com/deepseek-ai/DeepInfer) | Use cases: 4 | Page: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/ DeepInfer is an open-source, high-performance offline inference framework from DeepSeek for running DeepSeek models, currently DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Run offline inference on DeepSeek-V4.1-Flash locally: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/run-offline-inference-deepseek-v4-flash/ - Get repeatable model outputs for regression tests: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/get-repeatable-outputs-with-deterministic-sampling/ - Force model output to match a JSON schema: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/constrain-model-output-to-json-schema/ - Reproduce Deep Infer prefill and decode benchmarks: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/benchmark-prefill-and-decode-throughput/ ## How to run offline inference on DeepSeek-V4.1-Flash locally with Deep Infer Repo: [deepseek-ai/DeepInfer](https://github.com/deepseek-ai/DeepInfer) | Difficulty: Advanced | Time: 2 hours | Basis: From the README Build Deep Infer from source and generate text from the default DeepSeek checkpoint. **Who it is for:** An ML engineer with an SM100 GPU or Ascend 950 NPU who needs to test DeepSeek-V4.1-Flash outputs on their own hardware. **The problem:** Getting a DeepSeek model to run on a specific accelerator needs a build, the right environment variables, and a checkpoint in the right place. **The outcome:** A working Deep Infer build that loads DeepSeek-V4.1-Flash and produces generated text from the offline example script. ### Steps 1. Create and activate an empty Python 3.12 environment. 2. Run `export DEEPINFER_ACCELERATOR=nvidia` (use `ascend` instead on the Ascend platform), and keep it exported in the shell where you run Deep Infer. 3. Run `./build.sh` from the repo root. 4. (Optional) Run `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to use a local checkpoint instead of the Hugging Face cache. 5. Run `python examples/offline_inference.py` and read the output. ### Prompt to paste into your agent ```text Clone https://github.com/deepseek-ai/DeepInfer, set up a Python 3.12 environment, export DEEPINFER_ACCELERATOR=nvidia, run ./build.sh, then run python examples/offline_inference.py using the checkpoint at . Report the generated text and the build result, and verify that the script finishes without an import or device error. ``` ### Caveat The README says the open-source release covers offline inference only, and KV cache, prefill and decode disaggregation, and data-parallel load balancing are not included. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/run-offline-inference-deepseek-v4-flash/ ## How to get repeatable model outputs for regression tests with Deep Infer Repo: [deepseek-ai/DeepInfer](https://github.com/deepseek-ai/DeepInfer) | Difficulty: Advanced | Time: 2 hours | Basis: From the README Use batch-invariant inference and a fixed seed so identical inputs give identical outputs. **Who it is for:** A QA or ML engineer who needs to check whether a model change alters outputs and must rule out batch-related noise. **The problem:** Outputs can change with batch composition and sampling randomness, which makes before and after comparisons unreliable. **The outcome:** A test setup where the same input tokens and sampling parameters return the same output on every run on NVIDIA hardware. ### Steps 1. Run `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` in the shell that runs your inference script. 2. Pass a fixed `seq_seed` to `SamplingParams` in your script. 3. Run the same prompts twice with the same settings and save both outputs. 4. Compare the two saved outputs and record any difference as a failed check. ### Prompt to paste into your agent ```text In the DeepInfer repo, set DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, pass a fixed seq_seed to SamplingParams, and run twice. Output a diff of the two result files. Verify that the diff is empty. ``` ### Caveat The README says both batch-invariant inference and deterministic sampling are NVIDIA only, so this setup does not apply to Ascend. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/get-repeatable-outputs-with-deterministic-sampling/ ## How to force model output to match a JSON schema with Deep Infer Repo: [deepseek-ai/DeepInfer](https://github.com/deepseek-ai/DeepInfer) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Use structured outputs to limit generation to a regex, JSON schema, or EBNF grammar. **Who it is for:** A developer building a pipeline that needs the model to return machine-readable JSON with a fixed shape. **The problem:** Free-form model text is hard to parse, and retries on malformed output waste time. **The outcome:** Generated text that follows the schema you pass in, ready for a parser. ### Steps 1. Open `examples/_structured_output.py` and read how it builds the constraint. 2. Define your JSON schema as a Python dict. 3. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams` with your schema type and schema. 4. Run the offline example with your prompt and parse the returned text as JSON. ### Prompt to paste into your agent ```text In the DeepInfer repo, follow examples/_structured_output.py to constrain generation with so_args for the schema in . Output the generated JSON for the prompt . Verify that the output parses with json.loads and matches the schema's required keys. ``` ### Caveat The README does not list which schema types are supported beyond regex, JSON schema, and EBNF grammar, so check the example file before you rely on a specific type. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/constrain-model-output-to-json-schema/ ## How to reproduce Deep Infer prefill and decode benchmarks with Deep Infer Repo: [deepseek-ai/DeepInfer](https://github.com/deepseek-ai/DeepInfer) | Difficulty: Advanced | Time: 3 hours | Basis: From the README Run the sweep script to measure prefill and decode performance on SM100 GPUs. **Who it is for:** An infrastructure engineer comparing serving hardware or validating the published Deep Infer performance numbers. **The problem:** Published benchmark numbers are only useful if you can rerun them on your own setup. **The outcome:** Your own prefill and decode benchmark results that you can compare with the published charts. ### Steps 1. Complete the build steps so Deep Infer runs on your SM100 GPUs. 2. Read the README note that prefill was measured on 4 SM100 GPUs and decode on 64, each within one scale-up domain. 3. Run the sweep script at `examples/sweep/run_sweep.sh`. 4. Save the output and compare the prefill and decode results with the charts in the README. ### Prompt to paste into your agent ```text In the DeepInfer repo, run examples/sweep/run_sweep.sh on SM100 GPUs and save the output to . Report the prefill and decode results. Verify that the GPU count matches the layout the README describes before comparing with the published charts. ``` ### Caveat The published numbers used 4 GPUs for prefill and 64 for decode within one scale-up domain, so a smaller setup will not match the charts directly. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/deepseek-ai-deepinfer/benchmark-prefill-and-decode-throughput/