DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to check a Controller model on Type I questions with Loop Arena

Run one model on the 160 Type I Loop Contract questions without a Worker.

Short answer: Install Python 3.10+ and run python -m pip install -e '.[gateway]' from the repo root; then set your endpoint details with export OPENAI_API_KEY=YOUR_API_KEY and export OPENAI_BASE_URL=YOUR_BASE_URL, or copy the values from .env.example; run looparena-type1-run --data benchmarks/type1/questions --model MODEL_ID --preflight-only with your model ID in place of MODEL_ID. All steps.

  • Top pick
  • Research
  • Intermediate
  • 30 minutes
  • From the README
  • Any agent

Credentials. The setup uses an API key set in environment variables, so keep it out of committed files and shared logs.

Who it is forAn AI engineer who wants to see whether a candidate model can pick the right Loop Contract before a longer benchmark run.

The problemType II and III need Docker, task data, and many runs. A quick Type I check tells you if the model can answer the basic Controller decision.

The outcomeA JSONL output file with the model's answer to a single Type I question, after a preflight check of the endpoint and configuration.

What you need first

  • Python 3.10 or newer
  • An OpenAI-compatible model endpoint, its base URL, and an API key
  • The model ID you want to test
  • For Type II/III only: the pinned upstream task data, starting workspaces, and Docker runtimes described in docs/setup.md

Steps

  1. Install Python 3.10+ and run python -m pip install -e '.[gateway]' from the repo root.
  2. Set your endpoint details with export OPENAI_API_KEY=YOUR_API_KEY and export OPENAI_BASE_URL=YOUR_BASE_URL, or copy the values from .env.example.
  3. Run looparena-type1-run --data benchmarks/type1/questions --model MODEL_ID --preflight-only with your model ID in place of MODEL_ID. (inferred, check the README)
  4. Run looparena-type1-run --data benchmarks/type1/questions --model MODEL_ID --output runs/type1-smoke.jsonl --limit 1. (inferred, check the README)
  5. Open runs/type1-smoke.jsonl and confirm it contains one result for the chosen model.

Prompt for your agent

In the AMAP-ML/LoopArena repo, set OPENAI_API_KEY and OPENAI_BASE_URL from the environment, then run looparena-type1-run with --data benchmarks/type1/questions, --model <model ID>, and --preflight-only, followed by a one-question run with --limit 1 writing to runs/type1-smoke.jsonl. Report whether preflight passed and the path of the output file, then confirm the file holds exactly one result.

Caveat. The README gives the model ID only as a placeholder, so you must supply a model your endpoint actually serves.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for AMAP-ML/LoopArena and "Check a Controller model on Type I questions"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer