DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to benchmark a model on MMLU and TruthfulQA with Litgpt

Run Litgpt evaluate with the truthfulqa_mc2 and mmlu tasks to score a base or finetuned model.

Short answer: Install with pip install 'litgpt[extra]'; then run litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu' for the base model; run the same evaluate command on your finetuned checkpoint folder. All steps.

Who it is forAn ML engineer comparing a finetuned model with its base model.

The problemWithout a standard benchmark it is hard to tell if finetuning helped or hurt.

The outcomeBenchmark scores for the tasks you chose, ready to compare across checkpoints.

What you need first

  • Python with pip
  • A GPU for finetuning and pretraining
  • Disk space for model weights

Steps

  1. Install with pip install 'litgpt[extra]'
  2. Run litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu' for the base model
  3. Run the same evaluate command on your finetuned checkpoint folder
  4. Compare the scores per task

Prompt for your agent

Using Lightning-AI/litgpt, run `litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'` and the same command on <my checkpoint folder>, then put both sets of scores in one table.

Caveat. Full MMLU runs take a long time on small GPUs.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Lightning-AI/litgpt and "Benchmark a model on MMLU and TruthfulQA"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer