DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to evaluate a model you serve behind an API with LM Evaluation Harness

Point the harness at any OpenAI compatible endpoint, local or hosted, instead of loading weights.

Short answer: Install the API backend with pip install "lm_eval[api]"; then For a hosted model set your key, for example export OPENAI_API_KEY=YOUR_KEY_HERE, then run lm_eval --model openai-completions with --model_args model=davinci-002 and --tasks lambada_openai,hellaswag. All steps.

Credentials. The hosted route reads an API key from your environment, so keep it out of committed code.

cost. Every evaluated sample is a billed API call on a hosted model.

Who it is forA team that serves models with vLLM or a vendor API.

The problemLoading the model a second time just to evaluate it wastes a GPU.

The outcomeThe same benchmark numbers, measured against what you actually serve.

What you need first

  • Python with pip; the base package no longer includes transformers or torch
  • A model backend extra such as `pip install "lm_eval[hf]"`, `pip install "lm_eval[vllm]"` or `pip install "lm_eval[api]"`
  • A CUDA GPU for the local model examples, which pass --device cuda:0
  • An API key in the environment for hosted models

Steps

  1. Install the API backend with pip install "lm_eval[api]"
  2. For a hosted model set your key, for example export OPENAI_API_KEY=YOUR_KEY_HERE, then run lm_eval --model openai-completions with --model_args model=davinci-002 and --tasks lambada_openai,hellaswag
  3. For your own server run lm_eval --model local-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,batch_size=16
  4. Sanity check generative tasks with --limit 10 first, as the README recommends for chat APIs

Prompt for your agent

With EleutherAI/lm-evaluation-harness, evaluate the model behind <my vLLM endpoint> on gsm8k using the local-completions backend, starting with a 10 sample sanity run.

Caveat. Flags such as --device apply to local models only and do nothing for hosted ones.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for EleutherAI/lm-evaluation-harness and "Evaluate a model you serve behind an API"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer