# AI use cases for Halogen Server (peonist-ai/halogen-server)

Repo: [peonist-ai/halogen-server](https://github.com/peonist-ai/halogen-server) | Use cases: 4 | Page: https://dedupedai.com/use-cases/peonist-ai-halogen-server/

Halogen is a Python-based engine that runs the Qwen3.8-27B model on AMD Strix Halo (gfx1151) GPUs and serves an OpenAI-compatible API on port 8731.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Run a local OpenAI-compatible Qwen3.8-27B server: https://dedupedai.com/use-cases/peonist-ai-halogen-server/run-local-openai-endpoint-strix-halo/
- Measure prefill and decode speed on your own machine: https://dedupedai.com/use-cases/peonist-ai-halogen-server/benchmark-prefill-decode-on-own-hardware/
- Continue long chats without re-processing the context: https://dedupedai.com/use-cases/peonist-ai-halogen-server/resume-long-conversations-with-prompt-cache/
- Set a token budget that finishes reasoning and answers: https://dedupedai.com/use-cases/peonist-ai-halogen-server/tune-token-budget-for-reasoning-replies/

## How to run a local OpenAI-compatible Qwen3.8-27B server with Halogen Server

Repo: [peonist-ai/halogen-server](https://github.com/peonist-ai/halogen-server) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Start the halogen container on Strix Halo and call a local chat endpoint on port 8731.

**Who it is for:** A developer with an AMD Strix Halo machine who wants a private, local Qwen3.8-27B endpoint for their own apps or scripts.

**The problem:** Running a 27B model locally needs a serving setup tuned to the GPU, and generic runtimes do not target this hardware.

**The outcome:** An OpenAI-compatible chat endpoint answers requests on localhost port 8731 using the Qwen3.8-27B checkpoint.

### Steps

1. Install the Hugging Face CLI with `pip install -U "huggingface_hub[cli]"`.
2. Download the weights and tokenizer with `hf download peonist-ai/halogen-qwen3.8-27b --local-dir ~/halogen-models`. (inferred, check the README)
3. Start the server with `podman run --rm -p 8731:8731 --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host -v ~/halogen-models:/models:ro -v ~/halogen-models/tokenizer:/tokenizer:ro ghcr.io/peonist-ai/halogen:0.1.4`. (inferred, check the README)
4. Send a chat request with curl to http://localhost:8731/v1/chat/completions using a JSON body with a messages list. (inferred, check the README)
5. Read the reply's content field, and check finish_reason before trusting the answer.

### Prompt to paste into your agent

```text
In the repo peonist-ai/halogen-server, start the ghcr.io/peonist-ai/halogen:0.1.4 container with the checkpoint directory at <models_path> and the tokenizer at <models_path>/tokenizer, then send the prompt <prompt> to http://localhost:8731/v1/chat/completions. Expected output: a chat completion JSON with a non-empty message content. Verify that finish_reason is not "length".
```

### Caveat

The model weights are licensed separately from the engine by their original authors, and the image works only on Strix Halo (gfx1151) hardware.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/peonist-ai-halogen-server/run-local-openai-endpoint-strix-halo/

## How to measure prefill and decode speed on your own machine with Halogen Server

Repo: [peonist-ai/halogen-server](https://github.com/peonist-ai/halogen-server) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Run the bundled sweep and bench tools to get reproducible tokens-per-second figures.

**Who it is for:** A performance-minded engineer who needs real throughput numbers for their own Strix Halo box before choosing a local model setup.

**The problem:** Published speed figures for local models vary by prompt and configuration, so quoted numbers are hard to compare.

**The outcome:** A set of prefill and decode figures measured on the user's own hardware, with the prompt set and version named.

### Steps

1. Download the checkpoint and tokenizer as described for the endpoint setup.
2. Run the ten-shape benchmark with `podman run --rm --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host -v /path/to/models:/models:ro -v /path/to/tokenizer:/tokenizer:ro ghcr.io/peonist-ai/halogen:0.1.4 bench dflash2 256 low 3`. (inferred, check the README)
3. Run the llama-bench-shaped sweep with `podman run --rm ... ghcr.io/peonist-ai/halogen:0.1.4 sweep -p 512,2048,8192 -n 128,256 -d dflash2,mtp -r 3` (add the same device and volume flags as the bench command). (inferred, check the README)
4. Add `--json` to the sweep command if you want machine-readable output.
5. Record the version, the drafter, and the prompt set next to every number you publish.

### Prompt to paste into your agent

```text
Using the repo peonist-ai/halogen-server, run the bench and sweep commands from its README against the models at <models_path> on this machine. Expected output: prefill tokens per second for each prompt length and decode mean and range for each drafter. Verify that each reported figure names the version 0.1.4 and the prompt set.
```

### Caveat

Decode speed on this engine varies with the prompt type, so a single number is not meaningful unless the prompt set is named.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/peonist-ai-halogen-server/benchmark-prefill-decode-on-own-hardware/

## How to continue long chats without re-processing the context with Halogen Server

Repo: [peonist-ai/halogen-server](https://github.com/peonist-ai/halogen-server) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Use the prompt cache so follow-up turns on a long document resume instead of re-prefilling.

**Who it is for:** A user or developer running long-document Q&A or a long multi-turn session on a local Strix Halo server.

**The problem:** Each follow-up turn on a long context normally costs a full prefill, which dominates wait time at 32K tokens.

**The outcome:** Follow-up turns on a long conversation start quickly, with answers that match a cold run.

### Steps

1. Start the server with the default settings from the run command.
2. Send the long document in the first chat request to /v1/chat/completions on port 8731.
3. Send a follow-up request in the same conversation that reuses the same earlier messages.
4. Compare time to first token for the first and follow-up requests.
5. Set HALOGEN_CACHE_MB to 0 only if you want to disable the prompt cache for a comparison run.

### Prompt to paste into your agent

```text
In the repo peonist-ai/halogen-server, run the halogen:0.1.4 server, send a long document from <document_path> as the first chat turn, then send the follow-up question <question> in the same conversation. Expected output: a faster time to first token on the follow-up than on the first turn. Verify that the follow-up answer is the same as a cold run with the cache disabled.
```

### Caveat

The README gives roughly 20 times faster time to first token at 32K as a measured figure, so expect different gains on other prompt sizes.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/peonist-ai-halogen-server/resume-long-conversations-with-prompt-cache/

## How to set a token budget that finishes reasoning and answers with Halogen Server

Repo: [peonist-ai/halogen-server](https://github.com/peonist-ai/halogen-server) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Pick max_tokens so the model's thinking does not use up the whole reply budget.

**Who it is for:** A developer whose app gets empty replies from the local model because thinking tokens ran out the budget.

**The problem:** The model reasons before replying, and a budget that runs out mid-thought returns empty content with finish_reason set to length.

**The outcome:** Requests finish with a visible answer, and oversized budgets are rejected instead of silently cut.

### Steps

1. Read the per-request default of 8192 and the HALOGEN_MAX_TOKENS_CAP default of 65536 in the configuration table.
2. Send a request with max_completion_tokens, max_output_tokens, or max_tokens set to a larger value.
3. Check finish_reason in the reply, and read reasoning_content if content is empty.
4. Raise the budget in the request body if the reply shows finish_reason of length.
5. Set HALOGEN_MAX_TOKENS_CAP on the container if your app needs a different ceiling.

### Prompt to paste into your agent

```text
In the repo peonist-ai/halogen-server, send the prompt <prompt> to the local endpoint at http://localhost:8731/v1/chat/completions with a max_completion_tokens value of <tokens>. Expected output: a reply with finish_reason of stop and a non-empty content field. Verify that a budget above the cap returns a 400 error rather than a truncated answer.
```

### Caveat

The 8192 default was enough for every ordinary prompt the author measured, so longer reasoning tasks may still need a larger value.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/peonist-ai-halogen-server/tune-token-budget-for-reasoning-replies/