DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to use LM Evaluation Harness: what you can do with it

What is LM Evaluation Harness? The LM Evaluation Harness is EleutherAI's framework for few-shot evaluation of language models. One lm_eval command scores a model on named tasks, with backends for Hugging Face transformers, vLLM, SGLang, llama.cpp servers, ONNX Runtime, NeMo, Megatron and any OpenAI compatible API. It supports multi GPU evaluation, result saving, response logging and pushing results to the Hugging Face Hub.

How do I use it? Install the package with git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, cd lm-evaluation-harness and pip install -e; then add the backend with pip install "lm_eval[hf]"; List what you can run with lm-eval ls tasks. Full steps below.

1 person shared what they built with LM Evaluation Harness. See what people are building

Trended earlier

View repo on GitHub

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer

What do you want to do?

3 use cases

What people are building with LM Evaluation Harness

  • Benchmarkthe web

    Quantized a 3B reasoning model to 4-bit and scored it with lm-eval

    To fit VibeThinker-3B on a 6 GB laptop GPU he quantized it to 4-bit GPTQ, shrinking the weights from 5.8 GB to 2.0 GB, then served it in vLLM. He checked the reasoning survived with the harness on a 50 sample subset, reporting 84 percent on GSM8K against 70 percent for a comparable quantized Qwen2.5-3B, and flags the small sample himself.

    LM Evaluation Harness on GitHub

    By syedazeez on the web

    View original post on the web

Real builds shared on X, Hacker News, Reddit, DEV and GitHub, summed up by Deduped from each post and its links; each card links to the original post. See what people are building with other repos

Quick start with LM Evaluation Harness

  1. Check you have what it needs: Python with pip; the base package no longer includes transformers or torch; A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`; A CUDA GPU for the local model examples, which pass --device cuda:0.
  2. Open EleutherAI/lm-evaluation-harness on GitHub and read the README and license.
  3. Install the package with git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, cd lm-evaluation-harness and pip install -e .
  4. Add the backend with pip install "lm_eval[hf]"
  5. List what you can run with lm-eval ls tasks
  6. Evaluate with lm_eval --model hf plus --model_args pretrained=EleutherAI/gpt-j-6B, --tasks hellaswag, --device cuda:0 and --batch_size 8

These steps come from our top use case, Score a model on standard benchmarks in one command, which also has a copy-paste agent prompt.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for EleutherAI/lm-evaluation-harness

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Best AI repos for Software and DevOps

About this repo

Requirements

  • Python with pip; the base package no longer includes transformers or torch
  • A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`
  • A CUDA GPU for the local model examples, which pass --device cuda:0
  • An API key in the environment for hosted models

Add this badge to your README

Maintain LM Evaluation Harness? Show readers where to find these use cases. Use cases on Deduped

Markdown 

[![Use cases on Deduped](https://dedupedai.com/badge/EleutherAI/lm-evaluation-harness.svg)](https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/?utm_source=github&utm_medium=badge&utm_campaign=EleutherAI-lm-evaluation-harness)

HTML 

<a href="https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/?utm_source=github&amp;utm_medium=badge&amp;utm_campaign=EleutherAI-lm-evaluation-harness"><img src="https://dedupedai.com/badge/EleutherAI/lm-evaluation-harness.svg" alt="Use cases on Deduped" height="20"></a>

A static image, no tracking script. About the badge

Questions about LM Evaluation Harness

What is LM Evaluation Harness?

The LM Evaluation Harness is EleutherAI's framework for few-shot evaluation of language models. One lm_eval command scores a model on named tasks, with backends for Hugging Face transformers, vLLM, SGLang, llama.cpp servers, ONNX Runtime, NeMo, Megatron and any OpenAI compatible API. It supports multi GPU evaluation, result saving, response logging and pushing results to the Hugging Face Hub. It is an open-source project on GitHub: https://github.com/EleutherAI/lm-evaluation-harness.

How do I use LM Evaluation Harness?

Install the package with git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, cd lm-evaluation-harness and pip install -e; then add the backend with pip install "lm_eval[hf]"; List what you can run with lm-eval ls tasks. The use cases on this page give the full steps and a copy-paste agent prompt for each job.

Who is LM Evaluation Harness for?

ML engineers who need comparable benchmark numbers for the models they train, quantize or serve.

What can I do with LM Evaluation Harness?

3 use cases so far, for example: Score a model on standard benchmarks in one command; Evaluate a model you serve behind an API; Keep every eval run reproducible and shareable. Each has steps and a copy-paste agent prompt.

What do I need to run LM Evaluation Harness?

Python with pip; the base package no longer includes transformers or torch; A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`; A CUDA GPU for the local model examples, which pass --device cuda:0; An API key in the environment for hosted models.

What are alternatives to LM Evaluation Harness?

Other hot AI repos in Software and DevOps: ECC, Thinkrail, Superpowers, Deep Infer, Instructor.

Is LM Evaluation Harness safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.