# AI use cases for Eval (frontier-harness-eval/eval)

Repo: [frontier-harness-eval/eval](https://github.com/frontier-harness-eval/eval) | Use cases: 3 | Page: https://dedupedai.com/use-cases/frontier-harness-eval-eval/

A public results set and task definitions for comparing coding-agent harnesses that run the same Kimi K3 model on 30 software-engineering tasks, plus a workflow for evaluating your own harness.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Compare 12 coding-agent harness results side by side: https://dedupedai.com/use-cases/frontier-harness-eval-eval/browse-live-harness-results/
- Score your own coding harness on the 30-task set: https://dedupedai.com/use-cases/frontier-harness-eval-eval/evaluate-own-harness-on-published-tasks/
- Check if a harness score is comparable before citing it: https://dedupedai.com/use-cases/frontier-harness-eval-eval/check-result-comparability-before-citing/

## How to compare 12 coding-agent harness results side by side with Eval

Repo: [frontier-harness-eval/eval](https://github.com/frontier-harness-eval/eval) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README

Open the live FrontierHarness results to see pass rate, cost, and speed for each harness.

**Who it is for:** A developer picking a coding-agent harness who wants evidence before adopting one.

**The problem:** Vendor claims about coding agents rarely hold the model and tasks constant, so harness choice is hard to judge.

**The outcome:** You can see which harnesses passed the most tasks and which cost the least per pass, with the same model and tasks behind each row.

### Steps

1. Open the full results table in the README to see pass rate, median cost per pass, cache, and median time for each of the 12 configurations.
2. Open the interactive report at https://frontierharness.org to see failed runs, total cost per task, and task-level results.
3. Read the launch article at https://runta.com/blog/introducing-frontierharness-eval/ for the evaluation design and analysis.

### Prompt to paste into your agent

```text
Open the repo frontier-harness-eval/eval and read the Full results table in its README. Report the harness with the highest pass rate and the harness with the lowest median cost per pass, with their values. Verify that both values match the table row for each harness.
```

### Caveat

The published numbers come from one model and one set of 30 tasks, so the ranking may not carry over to your own work.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/frontier-harness-eval-eval/browse-live-harness-results/

## How to score your own coding harness on the 30-task set with Eval

Repo: [frontier-harness-eval/eval](https://github.com/frontier-harness-eval/eval) | Difficulty: Advanced | Time: 5 hours | Basis: From the README

Run the evaluation skill to get pass rate, cost, and a report for your harness.

**Who it is for:** A team that built a coding-agent harness and wants a like-for-like score against the published baselines.

**The problem:** Without a common task set and a fixed model, a harness score cannot be compared with published results.

**The outcome:** A normalized results file and HTML report for your harness, produced from one golden checkpoint with a fresh restore per task.

### Steps

1. Install the evaluation skill with `npx skills add frontier-harness-eval/eval --skill frontierharness-eval`, after installing the Runta companion skills with `npx skills add https://runta.com/docs --skill runta-installer runta-cli`.
2. Store your provider key once with `runta secret set FIREWORKS_API_KEY --prompt`, using the variable name for your provider.
3. Freeze the golden checkpoint with `$FH/provision-golden-checkpoint.sh` using your runtime, checkpoint, harness, provider, repo, and commit flags, where FH=skills/frontierharness-eval/scripts.
4. Run a small subset first with a file of suite-prefixed ids passed to `--tasks` to prove the plumbing, then run the full set with `$FH/run-trials.sh --checkpoint fh-golden-myharness-v1 --harness my-harness --provider fireworks --run-id 2026-09-02-myharness --out runs`. (inferred, check the README)
5. Score, chart, and build the report with `node $FH/normalize-results.mjs --run runs/2026-09-02-myharness --label "My Harness"`, then `node $FH/generate-chart.mjs --run runs/2026-09-02-myharness`, then `node $FH/build-report.mjs --run runs/2026-09-02-myharness`. (inferred, check the README)

### Prompt to paste into your agent

```text
Use the frontierharness-eval skill from the frontier-harness-eval/eval repo to evaluate <harness repo URL> on one Terminal-Bench task and one DeepSWE task first. Expected output: a normalized results file and a report under runs/<run-id>. Verify that run.json records egress_policy and that methodology_comparable is false for this new run.
```

### Caveat

New runs default to methodology_comparable false and get no leaderboard rank until you also run a control harness under the same policy and environment, and the 5-hour estimate is inferred from the per-task times in the table.

Risk, credentials: Your provider key is stored as a Runta secret and must never be pasted into shell history or the runtime, which should only see a stub value.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/frontier-harness-eval-eval/evaluate-own-harness-on-published-tasks/

## How to check if a harness score is comparable before citing it with Eval

Repo: [frontier-harness-eval/eval](https://github.com/frontier-harness-eval/eval) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Confirm the egress policy and comparability flag before you put a new score next to published ones.

**Who it is for:** A researcher or engineer about to publish or compare a new harness score against the Kimi K3 baselines.

**The problem:** Published baselines did not record their applied network allowlist, so a new score can look comparable when it is not.

**The outcome:** You know whether your run can be ranked, and what control run you still need to establish comparability.

### Steps

1. Open `results/eval-data.json` and `metadata/harness-versions.json` to see the published aggregate results and the harness versions used for the run.
2. In your own run folder, open `run.json` and read `egress_policy` for its mode, scope, and exact `allowed_hosts`.
3. Check that `methodology_comparable` is set to false for any new run that has not been compared against a control run.
4. Run a control harness under the same policy and environment, then compare the two runs before claiming comparability.

### Prompt to paste into your agent

```text
In the frontier-harness-eval/eval repo, read metadata/harness-versions.json and results/eval-data.json, then open the run.json in <run folder>. Report the egress_policy mode and whether methodology_comparable is true or false. Verify that the flag is false unless a matched control run is recorded.
```

### Caveat

The README says the published baselines did not record their applied allowlist, so no new score can be ranked against them without a matched control run.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/frontier-harness-eval-eval/check-result-comparability-before-citing/