Who it is forAn ML engineer comparing a finetuned model with its base model.
The problemWithout a standard benchmark it is hard to tell if finetuning helped or hurt.
The outcomeBenchmark scores for the tasks you chose, ready to compare across checkpoints.
What you need first
- Python with pip
- A GPU for finetuning and pretraining
- Disk space for model weights
Steps
- Install with
pip install 'litgpt[extra]' - Run
litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'for the base model - Run the same evaluate command on your finetuned checkpoint folder
- Compare the scores per task
Prompt for your agent
Using Lightning-AI/litgpt, run `litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'` and the same command on <my checkpoint folder>, then put both sets of scores in one table.
Caveat. Full MMLU runs take a long time on small GPUs.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for Lightning-AI/litgpt and "Benchmark a model on MMLU and TruthfulQA"
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeShare: X LinkedIn Hacker News Email Copy link
Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer