download. Model weights download on first serve.
Who it is forA developer whose app already uses the OpenAI client and wants a self hosted model.
The problemSwitching an app to a local model usually means rewriting its API calls.
The outcomeA local server at http://localhost:3000/v1 that the OpenAI client can call unchanged.
What you need first
- Python with pip
- A GPU sized for the model you pick
- A Hugging Face token for gated models
Steps
- Install with
pip install openllm - Set a Hugging Face token if the model is gated, as the README describes
- Start the server with
openllm serve llama3.2:1b - Create an OpenAI client with base_url http://localhost:3000/v1 and send a chat completion
- Swap the base URL in your own app to use the local model
Prompt for your agent
Using bentoml/OpenLLM, run `openllm serve llama3.2:1b`, then write a Python script with the OpenAI client and base_url 'http://localhost:3000/v1' that streams an answer to <my question>.
Caveat. Gated models such as Llama need a Hugging Face token before they download.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for bentoml/OpenLLM and "Serve Llama 3.2 as an OpenAI compatible API"
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeShare: X LinkedIn Hacker News Email Copy link
Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer