DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to benchmark decode and prefill on your own box with Halogen Flash Server

Reproduce the published throughput numbers on your own Strix Halo machine.

Who it is forA researcher or hardware tester who wants to check published speed claims against their own machine before trusting them.

The problemPublished throughput figures are only useful if they can be reproduced on other hardware, and the team wants comparable numbers.

The outcomeA set of prefill and decode measurements from the user's own machine, comparable with the figures in docs/performance.md.

What you need first

  • An AMD Strix Halo machine (the README measures on a Ryzen AI Max+ 395 with 128 GB)
  • Podman installed, with access to /dev/kfd and /dev/dri
  • Access to the BIOS to set the UMA frame buffer carve-out to its minimum
  • Disk space for the model weights, which the container downloads on first run
  • Network access to ghcr.io and the model download source

Steps

  1. Set up the server on the Strix Halo box using the Quickstart in docs/getting-started.md.
  2. Open docs/performance.md#served-throughput-end-to-end-over-http and follow the single command it gives for the prefill rows.
  3. Run the decode measurement the same page describes, using the kind of text each row names.
  4. Record the results with the host settings noted in docs/troubleshooting.md#the-host-settings-these-numbers-were-measured-on.
  5. Post the command and the curve to the Discord community linked in the README if you want to compare with others.

Prompt for your agent

Using the repo peonist-ai/halogen-flash-server, run the prefill and decode benchmarks described in docs/performance.md on <target machine>. Expected output: a table of prefill tok/s and decode tok/s for each prompt length and text type. Verify the host settings match docs/troubleshooting.md before comparing against the published 0.18.0 figures.

Caveat. The README notes that the best decode case uses short repetitive prompts, so the peak tok/s figure should not be read as typical output speed.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page