DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to watch tokens per second and TTFT on a local server with LLM Visuals

Attach the dashboard to a running llama-server and see tok/s, time to first token and GPU load update live.

Who it is forA developer running llama.cpp locally who wants to know how fast a model answers and whether the GPU is the bottleneck.

The problemA local server gives no live view of speed, latency or GPU use, so tuning a model setup is guesswork.

The outcomeA terminal dashboard that shows tokens per second, time to first token, GPU utilisation and memory for the server while a request runs.

What you need first

  • Rust toolchain 1.75 or newer
  • A C compiler, needed by the bundled SQLite used for --log-db
  • A locally listening llama-server for throughput panels (the demo mode needs none)
  • nvidia-smi for NVIDIA GPU panels, xpu-smi for Intel, the amdgpu driver for AMD, or macmon on Apple silicon

Steps

  1. Install a Rust toolchain (1.75 or newer), then run git clone https://github.com/DingoOz/llm-visuals.
  2. Run cd llm-visuals and then cargo run --release -- --demo to confirm the layout renders with synthetic data.
  3. Start your model with metrics enabled, for example llama-server --metrics, using the flag the README shows as llama-server.exe --metrics on Windows. (inferred, check the README)
  4. Run cargo run --release in the repo folder so the dashboard detects the running server.
  5. Press r to rescan if the server is not listed, then send a prompt and watch tok/s, TTFT and the GPU panels update.

Prompt for your agent

Using the repo DingoOz/llm-visuals, run `cargo run --release` against a llama-server listening on <server-port> with metrics enabled. Expected output: a dashboard listing that server with live tokens per second and time to first token while a request runs. Verify the model name and port shown match the server you started.

Caveat. Throughput panels need a locally listening llama-server, and GPU panels depend on the vendor tool for your hardware, so a machine without these shows only part of the dashboard.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page