# AI use cases for Agent Garten (MirroS-Lab/AgentGarten)

Repo: [MirroS-Lab/AgentGarten](https://github.com/MirroS-Lab/AgentGarten) | Use cases: 4 | Page: https://dedupedai.com/use-cases/mirros-lab-agentgarten/

AgentGarten is a Python repo with a neural renderer that turns depth and surface-normal data into video, plus training and streaming inference code; its code worlds and agent practice loop are not released yet.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Render comparison videos from a renderer checkpoint: https://dedupedai.com/use-cases/mirros-lab-agentgarten/render-validation-clips-from-checkpoint/
- Generate renderer video block by block with streaming: https://dedupedai.com/use-cases/mirros-lab-agentgarten/stream-renderer-blocks-with-kv-cache/
- Train the renderer on your own depth and normal clips: https://dedupedai.com/use-cases/mirros-lab-agentgarten/train-renderer-on-own-clip-dataset/
- Decode renderer RGB preview faster with TAEHV: https://dedupedai.com/use-cases/mirros-lab-agentgarten/speed-up-rgb-preview-with-taehv-decoder/

## How to render comparison videos from a renderer checkpoint with Agent Garten

Repo: [MirroS-Lab/AgentGarten](https://github.com/MirroS-Lab/AgentGarten) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Check a renderer checkpoint visually by writing validation videos without training.

**Who it is for:** An ML researcher who has a renderer checkpoint and needs to judge its visual quality before deciding what to do next.

**The problem:** Judging a checkpoint by loss numbers alone does not show whether the rendered video looks right against ground truth.

**The outcome:** Depth, normal, ground truth, and sample comparison videos written under samples/ for visual review.

### Steps

1. `git clone https://github.com/MirroS-Lab/AgentGarten.git`
2. `pip install -e '.[data,media,wandb,te]'`
3. `cp .env.example .env`
4. Run `hf download nvidia/Cosmos3-Nano --include 'transformer/*' 'text_tokenizer/*' 'assets/negative_prompt.json' --local-dir weights/Cosmos3-Nano` and `hf download Wan-AI/Wan2.2-TI2V-5B Wan2.2_VAE.pth --local-dir weights/Wan2.2-TI2V-5B`, then set WM_COSMOS3_NANO and WM_WAN22_VAE in .env to those paths. (inferred, check the README)
5. `bash scripts/run.sh experiment=cosmos3/bidirectional train.max_iterations=0 train.validate_at_start=true trainer.callbacks.samples.every_n=1` (inferred, check the README)

### Prompt to paste into your agent

```text
Using the AgentGarten repo at <repo path>, set up the environment, download the weights listed in the README, and render the validation clips of the checkpoint at <checkpoint path> without training. Expected output: comparison videos (depth, normal, ground truth, sample) under samples/. Verify that the samples/ folder contains one video set per validation clip before reporting success.
```

### Caveat

The README does not show how to load a specific checkpoint path directly, so the checkpoint may need to be selected through the training config (inferred).

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-agentgarten/render-validation-clips-from-checkpoint/

## How to generate renderer video block by block with streaming with Agent Garten

Repo: [MirroS-Lab/AgentGarten](https://github.com/MirroS-Lab/AgentGarten) | Difficulty: Advanced | Time: 2 hours 30 minutes | Basis: From the README

Run the renderer one block at a time with a bounded KV cache for step-by-step generation.

**Who it is for:** A developer building an interactive pipeline who needs the renderer's next block to depend on what was just generated.

**The problem:** Rendering a full clip in one pass does not let later conditions react to earlier output or keep memory bounded.

**The outcome:** A rendered sequence produced block by block, with each block committed to a bounded KV cache before the next one starts.

### Steps

1. `pip install -e '.[data,media,wandb,te]'`
2. Download the weights as described in the Weights section, then point WM_COSMOS3_NANO and WM_WAN22_VAE in .env at them.
3. Load the artifact with `load_artifact(inspect_artifact("weights/AgentGarten-renderer"))` and call `prepare_serving(artifact.student)`.
4. Create `Cosmos3Stream(artifact.student, StreamPolicy())`, call `stream.start(anchor, first_condition)`, then loop over blocks as shown in the Inference section.
5. Warm the input shapes once before serving, since kernel JIT and graph capture happen on first use.

### Prompt to paste into your agent

```text
Using the AgentGarten repo at <repo path>, write and run a script that streams the renderer block by block with Cosmos3Stream for the first-frame anchor at <anchor latent path>. Expected output: a sequence of committed blocks for the conditions in <conditions path>. Verify that the number of committed blocks equals the number of condition blocks before reporting success.
```

### Caveat

The README gives the streaming loop as a code sample with placeholder anchor and condition inputs, so you must build those latents yourself, and the real-time rendering engine and frontend are not released yet.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-agentgarten/stream-renderer-blocks-with-kv-cache/

## How to train the renderer on your own depth and normal clips with Agent Garten

Repo: [MirroS-Lab/AgentGarten](https://github.com/MirroS-Lab/AgentGarten) | Difficulty: Advanced | Time: 4 hours | Basis: From the README

Fine-tune the neural renderer on your own clips listed in a JSONL manifest.

**Who it is for:** A research team with its own video clips and matching depth and normal data who wants a renderer adapted to that domain.

**The problem:** A general renderer may not match the visual distribution of your own simulated or captured worlds.

**The outcome:** A renderer training run on your manifest, with output written under the configured output root.

### Steps

1. `pip install -e '.[data,media,wandb,te]'`
2. Prepare a JSONL manifest with one clip per line, using paths relative to the manifest, for example `{"video": "clips/0001.mp4", "depth": "clips/0001_depth.npy", "normal": "clips/0001_normal.mp4", "caption": "A car drives along a coastal road.", "depth_scale": 1.0}`. (inferred, check the README)
3. Set the manifest path and output root in .env, along with the weight paths from the Weights section.
4. `bash scripts/run.sh experiment=cosmos3/bidirectional`
5. For a multi-node run, set NNODES, NODE_RANK, and MASTER_ADDR, then run `bash scripts/run.sh experiment=cosmos3/bidirectional` on every node. (inferred, check the README)

### Prompt to paste into your agent

```text
Using the AgentGarten repo at <repo path>, create a JSONL manifest from the clips in <clips folder> with depth and normal files, then launch bidirectional training with scripts/run.sh. Expected output: training logs and checkpoints written under the output root in .env. Verify that the manifest has one valid line per clip and that the first training iteration starts before reporting success.
```

### Caveat

The README says the code worlds and agent practice loop will be released later, so this use case covers only renderer training on your own clips.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-agentgarten/train-renderer-on-own-clip-dataset/

## How to decode renderer RGB preview faster with TAEHV with Agent Garten

Repo: [MirroS-Lab/AgentGarten](https://github.com/MirroS-Lab/AgentGarten) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README

Attach the TAEHV decoder to speed up RGB output while encoding stays on the Wan encoder.

**Who it is for:** A researcher who reviews many renderer previews and finds RGB decoding slows each review cycle.

**The problem:** Full Wan decoding makes it slow to check many rendered clips during a review session.

**The outcome:** Faster RGB decoding for previews, with encoding unchanged on the Wan encoder.

### Steps

1. Complete the install and weight download steps from the Get started section.
2. Download `taew2_2_super.pth` from the TAEHV project at https://github.com/madebyollin/taehv to a local path. (inferred, check the README)
3. Add `model.conditioner.codec.decoder={_target_:wm.codecs.TAEW22SuperDecoder,pretrained_path:/path/to/taew2_2_super.pth}` as an extra override to the run command, replacing the path with yours.
4. Run the validation render from the first use case, for example `bash scripts/run.sh experiment=cosmos3/bidirectional train.max_iterations=0 train.validate_at_start=true trainer.callbacks.samples.every_n=1` with the override added. (inferred, check the README)

### Prompt to paste into your agent

```text
Using the AgentGarten repo at <repo path>, add the TAEHV decoder override for the file at <taew2_2_super.pth path> and render the validation clips. Expected output: the same comparison videos under samples/, with RGB decoded through the TAEHV decoder. Verify that the decoder path in the override exists on disk before reporting success.
```

### Caveat

The TAEHV decoder only changes RGB decoding, and the README states that encoding stays on the Wan encoder.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/mirros-lab-agentgarten/speed-up-rgb-preview-with-taehv-decoder/