DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to compare Controller models on paired coding tasks with Loop Arena

Run candidate Controllers on Type II and III coding tasks with a shared Worker.

Short answer: Read docs/protocol.md to confirm the role boundaries, budgets, and scoring; then follow docs/setup.md to pin the upstream source revisions, prepare the task data and starting workspaces, and install the Docker runtimes; set the Controller model and the shared Worker and Reporter model in the configuration described in docs/setup.md. All steps.

  • Research
  • Advanced
  • 4 hours
  • From the README
  • Any agent

Who it is forA coding-agent team that must decide which model should act as the runtime Controller for long repository edits.

The problemSingle-answer tests do not show whether a Controller can keep a Worker on track, verify progress, and stop at the right time across a full coding task.

The outcomeStrict Success Rate results for each candidate Controller across the 80 paired tasks, each repeated 3 times, using source-native terminal evaluation.

What you need first

  • Python 3.10 or newer
  • An OpenAI-compatible model endpoint, its base URL, and an API key
  • The model ID you want to test
  • For Type II/III only: the pinned upstream task data, starting workspaces, and Docker runtimes described in docs/setup.md

Steps

  1. Read docs/protocol.md to confirm the role boundaries, budgets, and scoring.
  2. Follow docs/setup.md to pin the upstream source revisions, prepare the task data and starting workspaces, and install the Docker runtimes.
  3. Set the Controller model and the shared Worker and Reporter model in the configuration described in docs/setup.md. (inferred, check the README)
  4. Run a single case first, then the full panel, using the single-case and panel run instructions in docs/setup.md.
  5. Compare the Strict Success Rate for each Controller against the Core-based scores in the README results table.

Prompt for your agent

In the AMAP-ML/LoopArena repo, follow docs/setup.md to prepare the Type II coding task data and Docker runtimes, then run one single-case Type II test with a <Controller model ID> as Controller and the shared Worker and Reporter from the setup guide. Report the Strict Success Rate for that case and the path of the run output, and verify that the protocol and budget validity checks passed.

Caveat. The README does not give setup times or resource needs, and the upstream repositories and container images must be obtained from their original sources, so the time estimate is rough.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for AMAP-ML/LoopArena and "Compare Controller models on paired coding tasks"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer