DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to score your own coding harness on the 30-task set with Eval

Run the evaluation skill to get pass rate, cost, and a report for your harness.

Credentials. Your provider key is stored as a Runta secret and must never be pasted into shell history or the runtime, which should only see a stub value.

Who it is forA team that built a coding-agent harness and wants a like-for-like score against the published baselines.

The problemWithout a common task set and a fixed model, a harness score cannot be compared with published results.

The outcomeA normalized results file and HTML report for your harness, produced from one golden checkpoint with a fresh restore per task.

What you need first

  • A Runta account, with the runta CLI installed and authenticated via runta login
  • jq, Node.js 18 or newer, and Python 3.9 or newer
  • A provider API key for Kimi K3 (FIREWORKS_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, or TOGETHER_API_KEY)
  • A harness repository you can build on a clean Linux box, plus an install script

Steps

  1. Install the evaluation skill with npx skills add frontier-harness-eval/eval --skill frontierharness-eval, after installing the Runta companion skills with npx skills add https://runta.com/docs --skill runta-installer runta-cli.
  2. Store your provider key once with runta secret set FIREWORKS_API_KEY --prompt, using the variable name for your provider.
  3. Freeze the golden checkpoint with $FH/provision-golden-checkpoint.sh using your runtime, checkpoint, harness, provider, repo, and commit flags, where FH=skills/frontierharness-eval/scripts.
  4. Run a small subset first with a file of suite-prefixed ids passed to --tasks to prove the plumbing, then run the full set with $FH/run-trials.sh --checkpoint fh-golden-myharness-v1 --harness my-harness --provider fireworks --run-id 2026-09-02-myharness --out runs. (inferred, check the README)
  5. Score, chart, and build the report with node $FH/normalize-results.mjs --run runs/2026-09-02-myharness --label "My Harness", then node $FH/generate-chart.mjs --run runs/2026-09-02-myharness, then node $FH/build-report.mjs --run runs/2026-09-02-myharness. (inferred, check the README)

Prompt for your agent

Use the frontierharness-eval skill from the frontier-harness-eval/eval repo to evaluate <harness repo URL> on one Terminal-Bench task and one DeepSWE task first. Expected output: a normalized results file and a report under runs/<run-id>. Verify that run.json records egress_policy and that methodology_comparable is false for this new run.

Caveat. New runs default to methodology_comparable false and get no leaderboard rank until you also run a control harness under the same policy and environment, and the 5-hour estimate is inferred from the per-task times in the table.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page