DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to check a finished run for missing scores before sharing with Harness Eval W

Verify that an evaluation run has every expected score before you publish a leaderboard row or report.

  • Research
  • Intermediate
  • 30 minutes
  • From the README
  • Any agent

Who it is forA benchmark maintainer or lab member who is about to share results and needs confirmation that no case was skipped or failed.

The problemA partial run can look complete in a table, so a missing score can slip into a published comparison.

The outcomeA clear pass or fail from the verify step, with the summary and leaderboard files confirmed before anyone shares them.

What you need first

  • Git and conda installed
  • Ability to create three conda environments from the files in docs/installation
  • Credentials and paths filled into harnesseval.env, copied from config/example.env
  • Generated results in the layout used by runs/example/results_example, or use the bundled demo

Steps

  1. conda activate harnesseval-main
  2. harnesseval verify run --eval-root runs/example/results_example/run/harnesseval/models/seedance-2.0-standard/evaluation --manifest runs/example/results_example/manifest.json --model seedance-2.0-standard (inferred: swap the --eval-root, --manifest, and --model values for your own run) (inferred, check the README)
  3. Open summary.json in the evaluation folder and confirm the overall and per-family scores are present.
  4. Open LEADERBOARD.md and leaderboard_latest.csv and check that the model row matches summary.json.

Prompt for your agent

In the HarnessEval-W repo, run harnesseval verify run with --eval-root set to <your-eval-root>, --manifest set to <your-manifest>, and --model set to <your-model-id>. Expect the verify command to confirm that all expected scores exist. Check that summary.json and LEADERBOARD.md agree on the overall score before you report the run as ready.

Caveat. The verify command only applies to a run that already exists, so you need a completed evaluation before this step works.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page