# AI use cases for Harness Eval W (MirroS-Lab/HarnessEval-W)

Repo: [MirroS-Lab/HarnessEval-W](https://github.com/MirroS-Lab/HarnessEval-W) | Use cases: 4 | Page: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/

HarnessEval-W is a Python evaluation pipeline that scores generated world model videos with sub-agents and saves a traceable reasoning chain for each score.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Score a world model's generated videos with evidence trees: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/score-world-model-videos-with-evidence-trees/
- Check a finished run for missing scores before sharing: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/check-finished-run-before-sharing-scores/
- Run the bundled demo to test the setup end to end: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/run-bundled-demo-to-test-setup/
- Submit a new case or skill to the benchmark: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/submit-new-case-or-skill-to-benchmark/

## How to score a world model's generated videos with evidence trees with Harness Eval W

Repo: [MirroS-Lab/HarnessEval-W](https://github.com/MirroS-Lab/HarnessEval-W) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Run the HarnessEval pipeline on a model's generated results and get scores backed by a saved reasoning trail.

**Who it is for:** A world model researcher who has generated rollout videos from a model and needs a scored, explainable result for each case.

**The problem:** Scalar benchmark scores hide why a rollout passed or failed, so physics or causality errors are hard to check.

**The outcome:** A run folder with summary.json, leaderboard files, and per-case artifacts that show the questions, answers, and scores behind each verdict.

### Steps

1. `git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git`
2. `cd harnesseval-w`
3. `conda env create -f docs/installation/main.environment.yml`, then `conda env create -f docs/installation/metrics.environment.yml`, then `conda env create -f docs/installation/pavrm.environment.yml`
4. `conda activate harnesseval-main`
5. `cp config/example.env harnesseval.env`, then `set -a; . ./harnesseval.env; set +a`
6. `harnesseval eval --results runs/example/results_example/generation --model-id seedance-2.0-standard --run-root runs/example/results_example/run --manifest runs/example/results_example/manifest.json --plan-root benchmark/plans` (inferred: replace the --results, --model-id, --run-root, and --manifest values with your own model's paths and id) (inferred, check the README)

### Prompt to paste into your agent

```text
Using the HarnessEval-W repo, set up the three conda environments from docs/installation, then run harnesseval eval with --results pointing to <your-generated-results-dir>, --model-id <your-model-id>, and the matching --run-root and --manifest paths. Expect a summary.json and LEADERBOARD.md under run/harnesseval/models/<your-model-id>/evaluation. Verify that summary.json lists an overall score and per-family scores before reporting back.
```

### Caveat

The README lists three separate conda environments and a credentials file, so the first run takes longer than a single install step.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/score-world-model-videos-with-evidence-trees/

## How to check a finished run for missing scores before sharing with Harness Eval W

Repo: [MirroS-Lab/HarnessEval-W](https://github.com/MirroS-Lab/HarnessEval-W) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Verify that an evaluation run has every expected score before you publish a leaderboard row or report.

**Who it is for:** A benchmark maintainer or lab member who is about to share results and needs confirmation that no case was skipped or failed.

**The problem:** A partial run can look complete in a table, so a missing score can slip into a published comparison.

**The outcome:** A clear pass or fail from the verify step, with the summary and leaderboard files confirmed before anyone shares them.

### Steps

1. `conda activate harnesseval-main`
2. `harnesseval verify run --eval-root runs/example/results_example/run/harnesseval/models/seedance-2.0-standard/evaluation --manifest runs/example/results_example/manifest.json --model seedance-2.0-standard` (inferred: swap the --eval-root, --manifest, and --model values for your own run) (inferred, check the README)
3. Open summary.json in the evaluation folder and confirm the overall and per-family scores are present.
4. Open LEADERBOARD.md and leaderboard_latest.csv and check that the model row matches summary.json.

### Prompt to paste into your agent

```text
In the HarnessEval-W repo, run harnesseval verify run with --eval-root set to <your-eval-root>, --manifest set to <your-manifest>, and --model set to <your-model-id>. Expect the verify command to confirm that all expected scores exist. Check that summary.json and LEADERBOARD.md agree on the overall score before you report the run as ready.
```

### Caveat

The verify command only applies to a run that already exists, so you need a completed evaluation before this step works.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/check-finished-run-before-sharing-scores/

## How to run the bundled demo to test the setup end to end with Harness Eval W

Repo: [MirroS-Lab/HarnessEval-W](https://github.com/MirroS-Lab/HarnessEval-W) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Use the included example results to confirm the install works before you bring your own model outputs.

**Who it is for:** A new user who has just installed Harness Eval W and wants proof the pipeline runs before using their own data.

**The problem:** Setup problems in three environments and the credentials file are hard to diagnose when a real model run fails.

**The outcome:** A completed demo run with scores in run/harnesseval/models/seedance-2.0-standard/evaluation and a successful verify step.

### Steps

1. `git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git`
2. `cd harnesseval-w`
3. `conda env create -f docs/installation/main.environment.yml`, then `conda env create -f docs/installation/metrics.environment.yml`, then `conda env create -f docs/installation/pavrm.environment.yml`
4. `conda activate harnesseval-main`
5. `cp config/example.env harnesseval.env`, then `set -a; . ./harnesseval.env; set +a`
6. `cd runs/example/results_example`
7. `harnesseval eval --results generation --model-id seedance-2.0-standard --run-root run --manifest manifest.json --plan-root ../../benchmark/plans`
8. `harnesseval verify run --eval-root run/harnesseval/models/seedance-2.0-standard/evaluation --manifest manifest.json --model seedance-2.0-standard`

### Prompt to paste into your agent

```text
In the HarnessEval-W repo, complete the setup from docs/installation, activate harnesseval-main, then run the bundled demo from runs/example/results_example with harnesseval eval and harnesseval verify run. Expect a summary.json under run/harnesseval/models/seedance-2.0-standard/evaluation. Verify that the verify command reports all scores present before saying the setup works.
```

### Caveat

The README says the demo can be evaluated immediately after setup, but the three conda environments still need to build first.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/run-bundled-demo-to-test-setup/

## How to submit a new case or skill to the benchmark with Harness Eval W

Repo: [MirroS-Lab/HarnessEval-W](https://github.com/MirroS-Lab/HarnessEval-W) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Draft a new world, action, or probe case, or a new evaluation skill, in the existing benchmark format for submission.

**Who it is for:** A researcher who has found a physics or causality failure the public set does not test and wants to contribute it.

**The problem:** Existing cases may not cover a useful kind of world model failure, and there is no shared place to propose one.

**The outcome:** A submission draft in the existing benchmark format, sent through the contribution page.

### Steps

1. Open the [Contribute page](https://mirros-lab.github.io/HarnessEval-W/contribute.html).
2. Decide whether you are submitting a new case (new world, action, or probe family) or a new skill (a new kind of evaluation).
3. Read the bundled benchmark files in the repo and match your submission to the existing format.
4. Write down the world, action, or probe intent and the evidence each sub-question needs.
5. Submit through the contribute page.

### Prompt to paste into your agent

```text
Read the HarnessEval-W repo and its public benchmark format, then draft a new <case or skill> for <the failure you want to test>. Expect a written draft with the world, action, probe intent, and sub-questions in the existing format. Check that the draft matches the format of the bundled examples before submitting it on the contribute page.
```

### Caveat

The README does not describe the exact submission template, so you need to read the bundled files to match the format.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-harnesseval-w/submit-new-case-or-skill-to-benchmark/