DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to serve an open LLM as a local HTTP API with Litgpt

Start Litgpt serve on an off the shelf model and call its /predict endpoint from Python.

Short answer: Install with pip install 'litgpt[extra]'; then List supported models with litgpt download list; start the server with litgpt serve microsoft/phi-2. All steps.

download. Model weights download on first use.

Who it is forA developer who wants a private model endpoint for an internal tool.

The problemHosted APIs send data off the machine and cost per call.

The outcomeA model server on 127.0.0.1:8000 that answers prompts posted to /predict.

What you need first

  • Python with pip
  • A GPU for finetuning and pretraining
  • Disk space for model weights

Steps

  1. Install with pip install 'litgpt[extra]'
  2. List supported models with litgpt download list
  3. Start the server with litgpt serve microsoft/phi-2
  4. From a separate Python session, POST a JSON body with a prompt to http://127.0.0.1:8000/predict using requests
  5. Read the reply from the output field of the JSON response

Prompt for your agent

With Lightning-AI/litgpt, run `litgpt serve microsoft/phi-2`, then write a small Python script that posts {"prompt": "Fix typos in the following sentence: Example input"} to http://127.0.0.1:8000/predict and prints the output field.

Caveat. The server listens on localhost only; add your own auth before exposing it.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Lightning-AI/litgpt and "Serve an open LLM as a local HTTP API"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer