DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Harness Eval W

HarnessEval-W is a Python evaluation pipeline that scores generated world model videos with sub-agents and saves a traceable reasoning chain for each score.

Trended earlier

MirroS-Lab/HarnessEval-W on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

4 use cases

  • Score a world model's generated videos with evidence trees

    Run the HarnessEval pipeline on a model's generated results and get scores backed by a saved reasoning trail.

    View prompt
    Using the HarnessEval-W repo, set up the three conda environments from docs/installation, then run harnesseval eval with --results pointing to <your-generated-results-dir>, --model-id <your-model-id>, and the matching --run-root and --manifest paths. Expect a summary.json and LEADERBOARD.md under run/harnesseval/models/<your-model-id>/evaluation. Verify that summary.json lists an overall score and per-family scores before reporting back.

  • Check a finished run for missing scores before sharing

    Verify that an evaluation run has every expected score before you publish a leaderboard row or report.

    View prompt
    In the HarnessEval-W repo, run harnesseval verify run with --eval-root set to <your-eval-root>, --manifest set to <your-manifest>, and --model set to <your-model-id>. Expect the verify command to confirm that all expected scores exist. Check that summary.json and LEADERBOARD.md agree on the overall score before you report the run as ready.

  • Run the bundled demo to test the setup end to end

    Use the included example results to confirm the install works before you bring your own model outputs.

    View prompt
    In the HarnessEval-W repo, complete the setup from docs/installation, activate harnesseval-main, then run the bundled demo from runs/example/results_example with harnesseval eval and harnesseval verify run. Expect a summary.json under run/harnesseval/models/seedance-2.0-standard/evaluation. Verify that the verify command reports all scores present before saying the setup works.

  • Submit a new case or skill to the benchmark

    Draft a new world, action, or probe case, or a new evaluation skill, in the existing benchmark format for submission.

    View prompt
    Read the HarnessEval-W repo and its public benchmark format, then draft a new <case or skill> for <the failure you want to test>. Expect a written draft with the world, action, probe intent, and sub-questions in the existing format. Check that the draft matches the format of the bundled examples before submitting it on the contribute page.

How to use Harness Eval W

  1. Check you have what it needs: Git and conda installed; Ability to create three conda environments from the files in docs/installation; Credentials and paths filled into harnesseval.env, copied from config/example.env.
  2. Open MirroS-Lab/HarnessEval-W on GitHub and read the README and license.
  3. git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git
  4. cd harnesseval-w
  5. conda env create -f docs/installation/main.environment.yml, then conda env create -f docs/installation/metrics.environment.yml, then conda env create -f docs/installation/pavrm.environment.yml
  6. conda activate harnesseval-main

These steps come from our top use case, Score a world model's generated videos with evidence trees, which also has a copy-paste agent prompt.

Harness Eval W alternatives · Best AI repos for Research

About this repo

Requirements

  • Git and conda installed
  • Ability to create three conda environments from the files in docs/installation
  • Credentials and paths filled into harnesseval.env, copied from config/example.env
  • Generated results in the layout used by runs/example/results_example, or use the bundled demo
Maintain Harness Eval W? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/mirros-lab-harnesseval-w.svg)](https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/?ref=badge)

More badge styles and HTML

Questions about Harness Eval W

What is Harness Eval W?

HarnessEval-W is a Python evaluation pipeline that scores generated world model videos with sub-agents and saves a traceable reasoning chain for each score. It is an open-source project on GitHub: https://github.com/MirroS-Lab/HarnessEval-W.

Who is Harness Eval W for?

Built for researchers and benchmark maintainers who compare world models and need scores they can audit.

What can I do with Harness Eval W?

4 use cases so far, for example: Score a world model's generated videos with evidence trees; Check a finished run for missing scores before sharing; Run the bundled demo to test the setup end to end. Each has steps and a copy-paste agent prompt.

What do I need to run Harness Eval W?

Git and conda installed; Ability to create three conda environments from the files in docs/installation; Credentials and paths filled into harnesseval.env, copied from config/example.env; Generated results in the layout used by runs/example/results_example, or use the bundled demo.

What are alternatives to Harness Eval W?

Other hot AI repos in Research: Scientific Agent Skills, Speaker Mem R1, DSH Council, Evo Bot, AI Engineering Course.

Is Harness Eval W safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.