DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Aa Agentperf Local

agentperf-local is an open-source tool that replays recorded agent conversations against a local OpenAI-compatible model server to measure throughput and latency.

Trended earlier

ArtificialAnalysis/aa-agentperf-local on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

5 use cases

  • Benchmark a local model server you already run

    Point the tool at a running OpenAI-compatible server and get tokens per second and latency numbers.

    View prompt
    Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local run` against the OpenAI-compatible server at <base-url> serving model <model-name>, writing output to results/<name>. Confirm that summary.json exists and report output tokens per second and median and p95 turn times. Verify the run is not marked reduced before you report it as comparable.

  • Download and benchmark a pinned catalog model

    Let the tool fetch a verified model, start llama.cpp, run the replay, and stop the server.

    View prompt
    Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local managed-run` with --profile-id qwen38-27b-q4-k-m and --framework llama-cpp, writing to results/<name>. Expect a summary.json and a server log in that directory. Verify that every model file passed its SHA-256 check and that the run is not marked reduced.

  • Check a setup quickly with the mini replay

    Confirm the tool and server work on a smaller device using the six-turn synthetic replay.

    View prompt
    Using the agentperf-local repo from ArtificialAnalysis, run the aa-mini-v1 replay with --context-tokens 8192 through managed-run on profile gemma4-12b-it-q4-0 and framework llama-cpp. Expect summary.json in results/<name>. Verify the output is marked reduced and state that it is not comparable with full-context results.

  • Benchmark Qwen on Splash on an Apple Silicon Mac

    Run the pinned Splash recipe on an M3 or newer Mac and get a verified, reproducible result.

    View prompt
    Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local managed-run` with --profile-id qwen38-27b-splash-dflash and --framework splash, writing to results/<name>. Expect summary.json in that directory. Verify that the recipe ran the exact policy and that the KV cache was eight-bit before reporting numbers.

  • Measure real tool time with live Docker tool calls

    Run the recorded shell commands in Docker containers so tool time is measured, not skipped.

    View prompt
    Using the agentperf-local repo from ArtificialAnalysis, build the default containers, then run `agentperf-local run` with --tool-mode live against <base-url> with model <model-name>, writing to results/<name>. Expect tools.json and summary.json in that directory. Verify that --live-network none was passed unless network access was explicitly needed.

How to use Aa Agentperf Local

  1. Check you have what it needs: uv installed (it fetches Python 3.12 if needed); A model server you already run with an OpenAI-compatible API, or a framework the tool can start (llama.cpp, vLLM, SGLang, or Splash on Apple Silicon); For full results, a server or device that can run a 65,536-token context at batch size 1.
  2. Open ArtificialAnalysis/aa-agentperf-local on GitHub and read the README and license.
  3. Install the tool with uv tool install agentperf-local
  4. Start your model server and note its base URL and model name, for example http://127.0.0.1:8080/v1 for llama.cpp
  5. Run agentperf-local run --base-url http://127.0.0.1:8080/v1 --model served-model --output-dir results/my-server (inferred, check the README)
  6. Open results/my-server/summary.json and read the headline numbers

These steps come from our top use case, Benchmark a local model server you already run, which also has a copy-paste agent prompt.

Aa Agentperf Local alternatives · Best AI repos for Software and DevOps

About this repo

Requirements

  • uv installed (it fetches Python 3.12 if needed)
  • A model server you already run with an OpenAI-compatible API, or a framework the tool can start (llama.cpp, vLLM, SGLang, or Splash on Apple Silicon)
  • For full results, a server or device that can run a 65,536-token context at batch size 1
  • Docker only if you want live tool calling
Maintain Aa Agentperf Local? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/artificialanalysis-aa-agentperf-local.svg)](https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/?ref=badge)

More badge styles and HTML

Questions about Aa Agentperf Local

What is Aa Agentperf Local?

agentperf-local is an open-source tool that replays recorded agent conversations against a local OpenAI-compatible model server to measure throughput and latency. It is an open-source project on GitHub: https://github.com/ArtificialAnalysis/aa-agentperf-local.

Who is Aa Agentperf Local for?

Built for people who run local LLM servers and want to know how fast their machine serves an AI agent, plus AI agents that run these benchmarks for them.

What can I do with Aa Agentperf Local?

5 use cases so far, for example: Benchmark a local model server you already run; Download and benchmark a pinned catalog model; Check a setup quickly with the mini replay. Each has steps and a copy-paste agent prompt.

What do I need to run Aa Agentperf Local?

uv installed (it fetches Python 3.12 if needed); A model server you already run with an OpenAI-compatible API, or a framework the tool can start (llama.cpp, vLLM, SGLang, or Splash on Apple Silicon); For full results, a server or device that can run a 65,536-token context at batch size 1; Docker only if you want live tool calling.

What are alternatives to Aa Agentperf Local?

Other hot AI repos in Software and DevOps: Harness2, iFixAi, Orca, Diagram Design, AI Memory.

Is Aa Agentperf Local safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.