DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to keep every eval run reproducible and shareable with LM Evaluation Harness

Write results and per sample responses to disk, cache repeat runs and push the results to the Hub.

Short answer: Install the harness and your backend extra as in the installation section; then add --output_path results to save the results of a run; add --log_samples to log model responses for post-hoc analysis. All steps.

Credentials. Pushing to the Hub needs an access token with write access in HF_TOKEN.

Who it is forA team tracking model quality over many runs.

The problemScores pasted into chat cannot be audited later.

The outcomeSaved results plus logged samples you can diff and share.

What you need first

  • Python with pip; the base package no longer includes transformers or torch
  • A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`
  • A CUDA GPU for the local model examples, which pass --device cuda:0
  • An API key in the environment for hosted models

Steps

  1. Install the harness and your backend extra as in the installation section
  2. Add --output_path results to save the results of a run
  3. Add --log_samples to log model responses for post-hoc analysis
  4. Speed up resumed runs with --use_cache <DIR>, and push to the Hub with --hf_hub_log_args after setting a write token in HF_TOKEN

Prompt for your agent

Using EleutherAI/lm-evaluation-harness, rerun my evaluation with --output_path and --log_samples, then show me how to diff two runs' sample outputs.

Caveat. Caching is rank dependent, so resume an interrupted run with the same GPU count.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for EleutherAI/lm-evaluation-harness and "Keep every eval run reproducible and shareable"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer