# AI use cases for Simreal ML Bench (Simreal-AI/Simreal-MLBench)

Repo: [Simreal-AI/Simreal-MLBench](https://github.com/Simreal-AI/Simreal-MLBench) | Use cases: 4 | Page: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/

Simreal-MLBench is a public Python package with a 60-task agentic ML research benchmark, its scoring code, and a client SDK, while the official evaluation runs on a Simreal-operated service.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Validate the 60-task benchmark catalog locally: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/validate-benchmark-catalog/
- Recompute a published benchmark score from saved counts: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/recompute-published-scores/
- Get an agent officially scored on the 60 tasks: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/evaluate-agent-via-simreal-service/
- Run the scoring test suite before changing the rule: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/check-scoring-code-with-tests/

## How to validate the 60-task benchmark catalog locally with Simreal ML Bench

Repo: [Simreal-AI/Simreal-MLBench](https://github.com/Simreal-AI/Simreal-MLBench) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Check the public task catalog and protocol config before building on the benchmark.

**Who it is for:** An ML engineer or benchmark maintainer who wants to confirm the public catalog is well-formed before adding an agent or a fork.

**The problem:** Benchmark definitions drift or break silently, and it is hard to know whether a local copy still matches the published protocol.

**The outcome:** A local confirmation that the task catalog and protocol configuration validate, with no credentials or model calls needed.

### Steps

1. Create a virtual environment with `python3 -m venv .venv`.
2. Activate it with `source .venv/bin/activate`.
3. Install the package and test extras with `python -m pip install -e '.[test]'`.
4. Validate the catalog with `mleb validate configs/catalog.json`.

### Prompt to paste into your agent

```text
In the Simreal-MLBench repository, create a virtual environment, install the package with `python -m pip install -e '.[test]'`, and run `mleb validate configs/catalog.json`. Report whether validation passed and how many tasks the catalog lists. Confirm the count is 60 before reporting success.
```

### Caveat

This only validates the public catalog and protocol config; it does not run any agent or touch the original competition datasets, which are acquired separately.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/validate-benchmark-catalog/

## How to recompute a published benchmark score from saved counts with Simreal ML Bench

Repo: [Simreal-AI/Simreal-MLBench](https://github.com/Simreal-AI/Simreal-MLBench) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Rebuild a reported task score from its reference counts to check the number yourself.

**Who it is for:** A research reviewer or lab analyst who has been given a reported ML Benchmark score and does not want to rely on trusting Simreal.

**The problem:** A published agent score is hard to trust without seeing how it was produced, and reviewers need to confirm the arithmetic and provenance.

**The outcome:** A recomputed task score from the public scoring code, plus a check of the submission id, artifact hash, and reference snapshot hash behind the number.

### Steps

1. Set up the environment with `python3 -m venv .venv`, `source .venv/bin/activate`, and `python -m pip install -e '.[test]'`.
2. Recompute a score from a snapshot and result pair with `mleb grade examples/snapshot.json examples/result.json`.
3. Run the report verification script with `python scripts/verify_report.py`.
4. Compare the submission id, artifact hash, and reference snapshot hash against the values in the reported result.
5. Check the arithmetic against the 100 × p² rule described in SCORING.md.

### Prompt to paste into your agent

```text
In the Simreal-MLBench repository, run `mleb grade examples/snapshot.json examples/result.json` and then `python scripts/verify_report.py`. Replace the example files with the saved reference counts for <target task> if you have them. Report the recomputed score and whether the verification script passed.
```

### Caveat

The local example files only demonstrate the grading path; checking a real reported number needs its saved reference counts, which the README says must be recomputed from saved data.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/recompute-published-scores/

## How to get an agent officially scored on the 60 tasks with Simreal ML Bench

Repo: [Simreal-AI/Simreal-MLBench](https://github.com/Simreal-AI/Simreal-MLBench) | Difficulty: Advanced | Time: 2 hours | Basis: From the README

Run an agent under the official protocol through the Simreal evaluation service.

**Who it is for:** An AI lab or team building a research agent that needs official, provenance-backed scores on the 60 ML Benchmark tasks.

**The problem:** Self-run agent results are hard to compare, and official scoring needs isolated workers, budget enforcement, and labels held outside the agent's reach.

**The outcome:** End-to-end agent runs under the official two-submission protocol, with official scores and full provenance records, arranged through Simreal.

### Steps

1. Read the Agent guide at AGENT_GUIDE.md and the Gateway guide at docs/GATEWAY.md.
2. Review the tier budgets: 6 hours for Easy, 12 hours for Medium, and 24 hours for Hard.
3. Integrate the agent with the public remote-client SDK (mleb-remote) as described in docs/GATEWAY.md.
4. Contact business@simreal.co to request evaluation access for the agent.
5. Once access is granted, run the agent through the service and keep the submission ids and artifact hashes for every reported number.

### Prompt to paste into your agent

```text
Read AGENT_GUIDE.md and docs/GATEWAY.md in the Simreal-MLBench repository. Summarize how an agent calls the mleb-remote SDK, the tier budgets, and the two-submission rule for <target agent>. Expected output: a short integration checklist. Verify that the checklist names the business contact before any run is planned.
```

### Caveat

Official evaluation runs only through the Simreal-operated service, which the README says is not in this repository, so access depends on contact with Simreal and the timing is not set by the repo.

Risk, credentials: The gateway uses a restricted forced-command SSH path and run identity, so keep any run credentials private and do not share them with other agents or teams.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/evaluate-agent-via-simreal-service/

## How to run the scoring test suite before changing the rule with Simreal ML Bench

Repo: [Simreal-AI/Simreal-MLBench](https://github.com/Simreal-AI/Simreal-MLBench) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Run the 35-test suite to confirm the scoring code still behaves as published.

**Who it is for:** A developer or researcher who plans to change, fork, or extend the scoring code and wants a baseline before editing it.

**The problem:** Changes to the scoring rule can quietly alter every published number, and it is hard to tell whether a fork still matches the public protocol.

**The outcome:** A passing baseline of the 35 scoring and verification tests, so any later change can be measured against it.

### Steps

1. Create a virtual environment with `python3 -m venv .venv`.
2. Activate it with `source .venv/bin/activate`.
3. Install the package and test extras with `python -m pip install -e '.[test]'`.
4. Run the test suite with `python -m pytest -q`.
5. Record the pass count so later scoring changes can be compared against it.

### Prompt to paste into your agent

```text
In the Simreal-MLBench repository, set up the virtual environment, install with `python -m pip install -e '.[test]'`, and run `python -m pytest -q`. Report the number of tests passed and any failures. Expected output: a pass count. Verify the count matches the 35 tests listed in the release status.
```

### Caveat

The 35-test baseline covers the public scoring and verification tooling only, not the Simreal-operated evaluation runtime or the official-scoring route.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/simreal-ai-simreal-mlbench/check-scoring-code-with-tests/