DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to score a harness against a hidden benchmark with Proteus

Attach a verifier as a hidden evaluator and keep only the harness changes that do not regress.

Short answer: pip install proteus-evolve==0.4.0; then write your verifier as a callable (trace, ctx) -> EvalResult and wrap it in EvaluatorSpec with Visibility.HIDDEN; Pass the spec to GoalConfig.of with selection="accept_reject". All steps.

Who it is forAn ML engineer with a benchmark verifier who wants a harness to improve on it without the agent seeing the score.

The problemWhen the agent can see the benchmark score, it may tune to the metric, and regressions can slip into the harness unnoticed.

The outcomeA goal configuration with a hidden evaluator and accept_reject selection, plus run output that measures each condition with the same ruler.

What you need first

  • Python 3.10 or newer
  • pip to install proteus-evolve==0.4.0 (no model SDK is required for the minimal harness)
  • For real harnesses: Docker and a DeepSeek or Codex key, as listed in the harness table

Steps

  1. pip install proteus-evolve==0.4.0
  2. Write your verifier as a callable (trace, ctx) -> EvalResult and wrap it in EvaluatorSpec with Visibility.HIDDEN
  3. Pass the spec to GoalConfig.of with selection="accept_reject"
  4. proteus check --harness mypkg.theirs_adapter:TheirsHarness --episode
  5. proteus run --harness mypkg.theirs_adapter:TheirsHarness --arm neutral ... (inferred, check the README)

Prompt for your agent

In the Proteus repo (proteus-evolve/Proteus), write a verifier callable that returns an EvalResult, wrap it in EvaluatorSpec with Visibility.HIDDEN, and run GoalConfig.of with selection="accept_reject" against the adapter at <harness path>. Expected output: a run folder with per-episode evaluator results and accepted or rejected snapshots. Verify that the hidden evaluator's score does not appear in any observe-phase prompt.

Caveat. This assumes you already have a working HarnessAdapter, and the README only shows the adapter path as a placeholder, so the check step must pass on your own adapter first.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for proteus-evolve/Proteus and "Score a harness against a hidden benchmark"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer