DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to score a model on standard benchmarks in one command with LM Evaluation Harness

Install the harness with the Hugging Face backend and evaluate a hub model on named tasks.

Short answer: Install the package with git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, cd lm-evaluation-harness and pip install -e; then add the backend with pip install "lm_eval[hf]"; List what you can run with lm-eval ls tasks. All steps.

  • Top pick
  • Software and DevOps
  • Intermediate
  • 2 hours
  • From the README
  • Claude Code
  • Codex
  • Cursor
  • Gemini CLI

cost. Benchmark runs can take hours of GPU time per model.

Who it is forAn engineer who needs a number to compare two checkpoints.

The problemEveryone's homemade eval script measures something slightly different.

The outcomeBenchmark scores for the tasks you named, on your own hardware.

What you need first

  • Python with pip; the base package no longer includes transformers or torch
  • A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`
  • A CUDA GPU for the local model examples, which pass --device cuda:0
  • An API key in the environment for hosted models

Steps

  1. Install the package with git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, cd lm-evaluation-harness and pip install -e .
  2. Add the backend with pip install "lm_eval[hf]"
  3. List what you can run with lm-eval ls tasks
  4. Evaluate with lm_eval --model hf plus --model_args pretrained=EleutherAI/gpt-j-6B, --tasks hellaswag, --device cuda:0 and --batch_size 8

Prompt for your agent

Using EleutherAI/lm-evaluation-harness, install it with the hf extra and score <my model> on hellaswag and lambada_openai, then report the numbers in a small table.

Caveat. Set --batch_size to auto if you are unsure what fits, as the README describes.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for EleutherAI/lm-evaluation-harness and "Score a model on standard benchmarks in one command"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer