DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Eval

A public results set and task definitions for comparing coding-agent harnesses that run the same Kimi K3 model on 30 software-engineering tasks, plus a workflow for evaluating your own harness.

Trended earlier

frontier-harness-eval/eval on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

3 use cases

  • Compare 12 coding-agent harness results side by side

    Open the live FrontierHarness results to see pass rate, cost, and speed for each harness.

    View prompt
    Open the repo frontier-harness-eval/eval and read the Full results table in its README. Report the harness with the highest pass rate and the harness with the lowest median cost per pass, with their values. Verify that both values match the table row for each harness.

  • Score your own coding harness on the 30-task set

    Run the evaluation skill to get pass rate, cost, and a report for your harness.

    View prompt
    Use the frontierharness-eval skill from the frontier-harness-eval/eval repo to evaluate <harness repo URL> on one Terminal-Bench task and one DeepSWE task first. Expected output: a normalized results file and a report under runs/<run-id>. Verify that run.json records egress_policy and that methodology_comparable is false for this new run.

  • Check if a harness score is comparable before citing it

    Confirm the egress policy and comparability flag before you put a new score next to published ones.

    View prompt
    In the frontier-harness-eval/eval repo, read metadata/harness-versions.json and results/eval-data.json, then open the run.json in <run folder>. Report the egress_policy mode and whether methodology_comparable is true or false. Verify that the flag is false unless a matched control run is recorded.

How to use Eval

  1. Check you have what it needs: A Runta account, with the runta CLI installed and authenticated via runta login; jq, Node.js 18 or newer, and Python 3.9 or newer; A provider API key for Kimi K3 (FIREWORKS_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, or TOGETHER_API_KEY).
  2. Open frontier-harness-eval/eval on GitHub and read the README and license.
  3. Open the full results table in the README to see pass rate, median cost per pass, cache, and median time for each of the 12 configurations.
  4. Open the interactive report at https://frontierharness.org to see failed runs, total cost per task, and task-level results.
  5. Read the launch article at https://runta.com/blog/introducing-frontierharness-eval/ for the evaluation design and analysis.

These steps come from our top use case, Compare 12 coding-agent harness results side by side, which also has a copy-paste agent prompt.

Eval alternatives · Best AI repos for Software and DevOps

About this repo

Requirements

  • A Runta account, with the runta CLI installed and authenticated via runta login
  • jq, Node.js 18 or newer, and Python 3.9 or newer
  • A provider API key for Kimi K3 (FIREWORKS_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, or TOGETHER_API_KEY)
  • A harness repository you can build on a clean Linux box, plus an install script
Maintain Eval? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/frontier-harness-eval-eval.svg)](https://dedupedai.com/use-cases/frontier-harness-eval-eval/?ref=badge)

More badge styles and HTML

Questions about Eval

What is Eval?

A public results set and task definitions for comparing coding-agent harnesses that run the same Kimi K3 model on 30 software-engineering tasks, plus a workflow for evaluating your own harness. It is an open-source project on GitHub: https://github.com/frontier-harness-eval/eval.

Who is Eval for?

Engineers and researchers choosing or building a coding-agent harness who need to compare pass rate and cost on the same model and tasks.

What can I do with Eval?

3 use cases so far, for example: Compare 12 coding-agent harness results side by side; Score your own coding harness on the 30-task set; Check if a harness score is comparable before citing it. Each has steps and a copy-paste agent prompt.

What do I need to run Eval?

A Runta account, with the runta CLI installed and authenticated via runta login; jq, Node.js 18 or newer, and Python 3.9 or newer; A provider API key for Kimi K3 (FIREWORKS_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, or TOGETHER_API_KEY); A harness repository you can build on a clean Linux box, plus an install script.

What are alternatives to Eval?

Other hot AI repos in Software and DevOps: Harness2, iFixAi, Orca, Diagram Design, AI Memory.

Is Eval safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.