# AI use cases for Rsi Eval (Danielxu0208/rsi_eval) Repo: [Danielxu0208/rsi_eval](https://github.com/Danielxu0208/rsi_eval) | Use cases: 4 | Page: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/ A lightweight Python SDK and CLI for evaluating recursive self-improvement methods across agent benchmarks. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Preview benchmark tasks before spending model quota: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/preview-benchmark-tasks-before-running/ - Run a Codex candidate on one Terminal-Bench task: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/run-codex-candidate-on-terminal-bench-task/ - Build your own self-improvement loop around evaluate(): https://dedupedai.com/use-cases/danielxu0208-rsi-eval/build-custom-rsi-loop-with-evaluate/ - Run a bundled RSI method on a prepared experiment: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/run-bundled-rsi-method-on-experiment/ ## How to preview benchmark tasks before spending model quota with Rsi Eval Repo: [Danielxu0208/rsi_eval](https://github.com/Danielxu0208/rsi_eval) | Difficulty: Intermediate | Time: 15 minutes | Basis: From the README Check which tasks, candidate hash, and settings a run would use, with no Docker or model calls. **Who it is for:** A developer or researcher who wants to confirm a benchmark selection before committing to a paid or slow evaluation. **The problem:** Benchmark runs can be large and consume model quota, so it is hard to verify task selection and settings first. **The outcome:** A printed preview of resolved task references, candidate hash, and run settings, without starting any execution. ### Steps 1. `git clone https://github.com/Danielxu0208/rsi_eval.git` 2. `cd rsi_eval` 3. `python3.12 -m venv .venv` 4. `source .venv/bin/activate` 5. `python -m pip install -e .` 6. `rsi-eval list` 7. `rsi-eval preview --config examples/evaluate.json` ### Prompt to paste into your agent ```text Clone https://github.com/Danielxu0208/rsi_eval, install it with `python -m pip install -e .` in a Python 3.12 virtual environment, then run `rsi-eval preview --config ` for the benchmark config I choose. Report the resolved task references and candidate hash. Verify that the preview ran without Docker or a model call. ``` ### Caveat Preview only confirms the plan; real evaluations still need Docker, the Harbor extra, and Codex setup described in the README. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/preview-benchmark-tasks-before-running/ ## How to run a Codex candidate on one Terminal-Bench task with Rsi Eval Repo: [Danielxu0208/rsi_eval](https://github.com/Danielxu0208/rsi_eval) | Difficulty: Advanced | Time: 180 minutes | Basis: From the README Score a candidate agent on a single Terminal-Bench 2.0 task and read the per-task result. **Who it is for:** A software engineer testing a coding agent setup who needs a real score on one task before running a larger suite. **The problem:** Setting up Docker, Harbor, and a dedicated Codex login is easy to get wrong, and a first run can be hard to diagnose. **The outcome:** A result.json and resolved.json for the candidate, plus a reference oracle run to compare against. ### Steps 1. `python -m pip install -e '.[harbor]'` 2. `docker info` 3. `CODEX_HOME="$HOME/.codex-rsi-eval" codex -c 'cli_auth_credentials_store="file"' login` (inferred, check the README) 4. `rsi-eval configure --model YOUR_MODEL --auth-dir "$HOME/.codex-rsi-eval" --output profile.json` (inferred, check the README) 5. `rsi-eval check --config examples/evaluate.json` 6. `rsi-eval run --config examples/evaluate.json --output runs/tb20-candidate` (inferred, check the README) ### Prompt to paste into your agent ```text In the rsi_eval repo, set target.model and the Codex login directory in examples/evaluate.json for , then run `rsi-eval run --config examples/evaluate.json --output runs/`. Report status and per-task reward from result.json. Verify that the output directory did not exist before the run. ``` ### Caveat The candidate run consumes model quota, and the README says the bundled subscription Codex target requires concurrency=1. Risk, credentials: The Codex login directory and the profile hold account access, so keep them outside the repository and out of candidate artifacts. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/run-codex-candidate-on-terminal-bench-task/ ## How to build your own self-improvement loop around evaluate() with Rsi Eval Repo: [Danielxu0208/rsi_eval](https://github.com/Danielxu0208/rsi_eval) | Difficulty: Advanced | Time: 240 minutes | Basis: From the README Call evaluate() from your own loop to test each new candidate and feed scores into the next one. **Who it is for:** A researcher developing a recursive self-improvement method who wants to keep control of candidate generation, memory, and selection. **The problem:** Evaluating each candidate across benchmarks means writing execution and result handling for every task, which takes time away from the method itself. **The outcome:** A Python loop that generates a candidate, scores it on chosen tasks, and uses per-task feedback to produce the next candidate. ### Steps 1. `python -m pip install -e '.[harbor]'` 2. Write a Python script that imports TargetSpec, BenchmarkSelection, RunOptions, and evaluate from Rsi Eval, following the SDK example in the README. 3. In the loop, build a TargetSpec from your candidate directory and call evaluate() with a new output_dir for each candidate. 4. Read result.status and each trial's task_id, status, and reward, and pass them to your candidate generator. ### Prompt to paste into your agent ```text Using the rsi_eval SDK, write a loop that generates a candidate in , calls evaluate() with a new output directory per candidate, and collects per-task rewards. Report the reward for each candidate across iterations. Verify that each evaluation wrote its own result.json and resolved.json. ``` ### Caveat The README notes that bundled Codex targets need concurrency=1, and each loop iteration with a model consumes quota. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/build-custom-rsi-loop-with-evaluate/ ## How to run a bundled RSI method on a prepared experiment with Rsi Eval Repo: [Danielxu0208/rsi_eval](https://github.com/Danielxu0208/rsi_eval) | Difficulty: Advanced | Time: 300 minutes | Basis: From the README Prepare, check, run, and independently test a Meta-Harness, RRSI, AHE, or HarnessX workflow. **Who it is for:** A researcher who wants a ready-made self-improvement workflow instead of writing one from scratch. **The problem:** Each method has its own dependencies, pinned upstream code, and seed handling, which are tedious to set up by hand. **The outcome:** An experiment directory with a frozen configuration, a method run, and an independent test of the selected artifact. ### Steps 1. `rsi-eval method list` 2. `rsi-eval method prepare --method meta-harness --profile profile.json --experiment runs/meta-example` (inferred, check the README) 3. `rsi-eval method check --experiment runs/meta-example` 4. `rsi-eval method run --experiment runs/meta-example` 5. `rsi-eval method test --experiment runs/meta-example --output runs/meta-test` ### Prompt to paste into your agent ```text In the rsi_eval repo, prepare a experiment at runs/ using profile.json, then run method check, run, and test in that order. Report the artifact selected by the method and the test output. Verify that the experiment directory contains the frozen configuration and seed. ``` ### Caveat The README says prepare downloads packages and sources without calling a model, but the run step does call models and consumes quota. Risk, credentials: The method runs use the Codex profile and login directory, so keep those credentials out of the repository and candidate artifacts. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/danielxu0208-rsi-eval/run-bundled-rsi-method-on-experiment/