# AI use cases for LM Evaluation Harness (EleutherAI/lm-evaluation-harness) Repo: [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Use cases: 3 | Page: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/ The LM Evaluation Harness is EleutherAI's framework for few-shot evaluation of language models. One lm_eval command scores a model on named tasks, with backends for Hugging Face transformers, vLLM, SGLang, llama.cpp servers, ONNX Runtime, NeMo, Megatron and any OpenAI compatible API. It supports multi GPU evaluation, result saving, response logging and pushing results to the Hugging Face Hub. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Score a model on standard benchmarks in one command: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/benchmark-a-hugging-face-model-with-lm-eval/ - Evaluate a model you serve behind an API: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/evaluate-a-served-model-through-its-openai-api/ - Keep every eval run reproducible and shareable: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/save-and-share-lm-eval-results/ ## How to score a model on standard benchmarks in one command with LM Evaluation Harness Repo: [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README Install the harness with the Hugging Face backend and evaluate a hub model on named tasks. **Who it is for:** An engineer who needs a number to compare two checkpoints. **The problem:** Everyone's homemade eval script measures something slightly different. **The outcome:** Benchmark scores for the tasks you named, on your own hardware. ### Steps 1. Install the package with `git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness`, `cd lm-evaluation-harness` and `pip install -e .` 2. Add the backend with `pip install "lm_eval[hf]"` 3. List what you can run with `lm-eval ls tasks` 4. Evaluate with `lm_eval --model hf` plus `--model_args pretrained=EleutherAI/gpt-j-6B`, `--tasks hellaswag`, `--device cuda:0` and `--batch_size 8` ### Prompt to paste into your agent ```text Using EleutherAI/lm-evaluation-harness, install it with the hf extra and score on hellaswag and lambada_openai, then report the numbers in a small table. ``` ### Caveat Set --batch_size to auto if you are unsure what fits, as the README describes. Risk, cost: Benchmark runs can take hours of GPU time per model. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/benchmark-a-hugging-face-model-with-lm-eval/ ## How to evaluate a model you serve behind an API with LM Evaluation Harness Repo: [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README Point the harness at any OpenAI compatible endpoint, local or hosted, instead of loading weights. **Who it is for:** A team that serves models with vLLM or a vendor API. **The problem:** Loading the model a second time just to evaluate it wastes a GPU. **The outcome:** The same benchmark numbers, measured against what you actually serve. ### Steps 1. Install the API backend with `pip install "lm_eval[api]"` 2. For a hosted model set your key, for example `export OPENAI_API_KEY=YOUR_KEY_HERE`, then run `lm_eval --model openai-completions` with `--model_args model=davinci-002` and `--tasks lambada_openai,hellaswag` 3. For your own server run `lm_eval --model local-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,batch_size=16` 4. Sanity check generative tasks with `--limit 10` first, as the README recommends for chat APIs ### Prompt to paste into your agent ```text With EleutherAI/lm-evaluation-harness, evaluate the model behind on gsm8k using the local-completions backend, starting with a 10 sample sanity run. ``` ### Caveat Flags such as --device apply to local models only and do nothing for hosted ones. Risk, credentials: The hosted route reads an API key from your environment, so keep it out of committed code. Risk, cost: Every evaluated sample is a billed API call on a hosted model. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/evaluate-a-served-model-through-its-openai-api/ ## How to keep every eval run reproducible and shareable with LM Evaluation Harness Repo: [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Write results and per sample responses to disk, cache repeat runs and push the results to the Hub. **Who it is for:** A team tracking model quality over many runs. **The problem:** Scores pasted into chat cannot be audited later. **The outcome:** Saved results plus logged samples you can diff and share. ### Steps 1. Install the harness and your backend extra as in the installation section 2. Add `--output_path results` to save the results of a run 3. Add `--log_samples` to log model responses for post-hoc analysis 4. Speed up resumed runs with `--use_cache `, and push to the Hub with `--hf_hub_log_args` after setting a write token in HF_TOKEN ### Prompt to paste into your agent ```text Using EleutherAI/lm-evaluation-harness, rerun my evaluation with --output_path and --log_samples, then show me how to diff two runs' sample outputs. ``` ### Caveat Caching is rank dependent, so resume an interrupted run with the same GPU count. Risk, credentials: Pushing to the Hub needs an access token with write access in HF_TOKEN. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/eleutherai-lm-evaluation-harness/save-and-share-lm-eval-results/