# AI use cases for Aa Agentperf Local (ArtificialAnalysis/aa-agentperf-local)

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Use cases: 5 | Page: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/

agentperf-local is an open-source tool that replays recorded agent conversations against a local OpenAI-compatible model server to measure throughput and latency.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Benchmark a local model server you already run: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/benchmark-attached-local-server/
- Download and benchmark a pinned catalog model: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/managed-run-pinned-catalog-model/
- Check a setup quickly with the mini replay: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/quick-install-check-small-context/
- Benchmark Qwen on Splash on an Apple Silicon Mac: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/benchmark-splash-on-apple-silicon/
- Measure real tool time with live Docker tool calls: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/live-tool-calling-benchmark/

## How to benchmark a local model server you already run with Aa Agentperf Local

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Point the tool at a running OpenAI-compatible server and get tokens per second and latency numbers.

**Who it is for:** A developer who already runs llama.cpp, LM Studio, vLLM, SGLang, or Ollama locally and wants to know how fast it serves agent-style requests.

**The problem:** Chat tests do not show how a server handles long agent conversations where every request carries the full history.

**The outcome:** A results directory with summary.json holding output tokens per second and median and p95 first-token and turn times.

### Steps

1. Install the tool with `uv tool install agentperf-local`
2. Start your model server and note its base URL and model name, for example http://127.0.0.1:8080/v1 for llama.cpp
3. Run `agentperf-local run --base-url http://127.0.0.1:8080/v1 --model served-model --output-dir results/my-server` (inferred, check the README)
4. Open results/my-server/summary.json and read the headline numbers

### Prompt to paste into your agent

```text
Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local run` against the OpenAI-compatible server at <base-url> serving model <model-name>, writing output to results/<name>. Confirm that summary.json exists and report output tokens per second and median and p95 turn times. Verify the run is not marked reduced before you report it as comparable.
```

### Caveat

The exact output policy sends ignore_eos, and the tool stops if your server does not honor it. Ollama cannot honor it, so Ollama results use the recorded policy and are not directly comparable.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/benchmark-attached-local-server/

## How to download and benchmark a pinned catalog model with Aa Agentperf Local

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Let the tool fetch a verified model, start llama.cpp, run the replay, and stop the server.

**Who it is for:** A local AI enthusiast or hardware reviewer who wants a reproducible benchmark of a specific model on their own machine.

**The problem:** Setting up a model server, pinning the exact model files, and running a comparable replay by hand is error-prone.

**The outcome:** A results directory with the replay summary, plus the server log and deployment record written by the managed run.

### Steps

1. Install the tool with `uv tool install agentperf-local`
2. Make sure `llama-server` is on your PATH
3. Run `agentperf-local managed-run --profile-id qwen38-27b-q4-k-m --framework llama-cpp --output-dir results/qwen38-27b` (inferred, check the README)
4. Check summary.json and the server log in results/qwen38-27b

### Prompt to paste into your agent

```text
Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local managed-run` with --profile-id qwen38-27b-q4-k-m and --framework llama-cpp, writing to results/<name>. Expect a summary.json and a server log in that directory. Verify that every model file passed its SHA-256 check and that the run is not marked reduced.
```

### Caveat

A full comparable run needs a 65,536-token context at batch size 1, so a device that cannot fit it can only produce a reduced run that is not comparable with full results.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/managed-run-pinned-catalog-model/

## How to check a setup quickly with the mini replay with Aa Agentperf Local

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Confirm the tool and server work on a smaller device using the six-turn synthetic replay.

**Who it is for:** A user with limited memory who wants to confirm the install works before attempting a full benchmark.

**The problem:** The default replay needs about 58,000 tokens of context, which many machines cannot fit.

**The outcome:** A short reduced-context run that verifies the setup. Its results are marked as not comparable.

### Steps

1. Install the tool with `uv tool install agentperf-local`
2. Run `agentperf-local managed-run --profile-id gemma4-12b-it-q4-0 --framework llama-cpp --replay aa-mini-v1 --context-tokens 8192 --output-dir results/gemma4-12b-mini` (inferred, check the README)
3. Confirm the run finished and that summary.json exists in results/gemma4-12b-mini

### Prompt to paste into your agent

```text
Using the agentperf-local repo from ArtificialAnalysis, run the aa-mini-v1 replay with --context-tokens 8192 through managed-run on profile gemma4-12b-it-q4-0 and framework llama-cpp. Expect summary.json in results/<name>. Verify the output is marked reduced and state that it is not comparable with full-context results.
```

### Caveat

The aa-mini-v1 replay is an install check only, and its results are not comparable with the default replay.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/quick-install-check-small-context/

## How to benchmark Qwen on Splash on an Apple Silicon Mac with Aa Agentperf Local

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Run the pinned Splash recipe on an M3 or newer Mac and get a verified, reproducible result.

**Who it is for:** A Mac owner with an M3 or newer chip on macOS 26.4 or later who wants to measure Splash performance.

**The problem:** Serving a large model on a Mac with a custom engine means pinning versions and model snapshots so results can be compared.

**The outcome:** A results directory for the Splash recipe, with the snapshot verified and the run using the exact output policy.

### Steps

1. Install the Splash engine with `brew install incoai/tap/splash`
2. Install the tool with `uv tool install agentperf-local`
3. Run `agentperf-local managed-run --profile-id qwen38-27b-splash-dflash --framework splash --output-dir results/qwen38-splash` (inferred, check the README)
4. Read summary.json in results/qwen38-splash

### Prompt to paste into your agent

```text
Using the agentperf-local repo from ArtificialAnalysis, run `agentperf-local managed-run` with --profile-id qwen38-27b-splash-dflash and --framework splash, writing to results/<name>. Expect summary.json in that directory. Verify that the recipe ran the exact policy and that the KV cache was eight-bit before reporting numbers.
```

### Caveat

The recipe needs an M3 or newer Mac on macOS 26.4 or later, so it will not run on other hardware.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/benchmark-splash-on-apple-silicon/

## How to measure real tool time with live Docker tool calls with Aa Agentperf Local

Repo: [ArtificialAnalysis/aa-agentperf-local](https://github.com/ArtificialAnalysis/aa-agentperf-local) | Difficulty: Advanced | Time: 2 hours | Basis: From the README

Run the recorded shell commands in Docker containers so tool time is measured, not skipped.

**Who it is for:** A researcher or infrastructure engineer who needs agent timings that include real tool execution, not sleep estimates.

**The problem:** By default the replay skips tool time between turns, so end-to-end agent timing can be understated.

**The outcome:** A run with live tool timing and a results directory that includes tools.json.

### Steps

1. Run `scripts/install-swebench-validation.sh` on arm64 hosts only, once
2. Run `scripts/build-default-containers.sh`
3. Run `uv run agentperf-local run --base-url http://127.0.0.1:8080/v1 --model served-model --output-dir results/live-tools --tool-mode live` (inferred, check the README)
4. Read tools.json and summary.json in results/live-tools

### Prompt to paste into your agent

```text
Using the agentperf-local repo from ArtificialAnalysis, build the default containers, then run `agentperf-local run` with --tool-mode live against <base-url> with model <model-name>, writing to results/<name>. Expect tools.json and summary.json in that directory. Verify that --live-network none was passed unless network access was explicitly needed.
```

### Caveat

Docker is not a security boundary and access to the Docker daemon is equivalent to root on most hosts, so review the security caveats before running.

Risk, remote script: The build scripts pull or build third-party images and clone the SWE-bench harness from GitHub, and the recorded shell commands are untrusted input.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/artificialanalysis-aa-agentperf-local/live-tool-calling-benchmark/