download. Model weights download on first use.
Who it is forA developer who wants a private model endpoint for an internal tool.
The problemHosted APIs send data off the machine and cost per call.
The outcomeA model server on 127.0.0.1:8000 that answers prompts posted to /predict.
What you need first
- Python with pip
- A GPU for finetuning and pretraining
- Disk space for model weights
Steps
- Install with
pip install 'litgpt[extra]' - List supported models with
litgpt download list - Start the server with
litgpt serve microsoft/phi-2 - From a separate Python session, POST a JSON body with a prompt to http://127.0.0.1:8000/predict using requests
- Read the reply from the output field of the JSON response
Prompt for your agent
With Lightning-AI/litgpt, run `litgpt serve microsoft/phi-2`, then write a small Python script that posts {"prompt": "Fix typos in the following sentence: Example input"} to http://127.0.0.1:8000/predict and prints the output field.
Caveat. The server listens on localhost only; add your own auth before exposing it.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for Lightning-AI/litgpt and "Serve an open LLM as a local HTTP API"
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeShare: X LinkedIn Hacker News Email Copy link
Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer