DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to serve Llama 3.2 as an OpenAI compatible API with OpenLLM

Run openllm serve on llama3.2:1b and point the OpenAI Python client at localhost:3000/v1.

Short answer: Install with pip install openllm; then set a Hugging Face token if the model is gated, as the README describes; start the server with openllm serve llama3.2:1b. All steps.

  • Top pick
  • Software and DevOps
  • Beginner
  • 20 minutes
  • From the README
  • Claude Code
  • Codex
  • Cursor
  • Gemini CLI

download. Model weights download on first serve.

Who it is forA developer whose app already uses the OpenAI client and wants a self hosted model.

The problemSwitching an app to a local model usually means rewriting its API calls.

The outcomeA local server at http://localhost:3000/v1 that the OpenAI client can call unchanged.

What you need first

  • Python with pip
  • A GPU sized for the model you pick
  • A Hugging Face token for gated models

Steps

  1. Install with pip install openllm
  2. Set a Hugging Face token if the model is gated, as the README describes
  3. Start the server with openllm serve llama3.2:1b
  4. Create an OpenAI client with base_url http://localhost:3000/v1 and send a chat completion
  5. Swap the base URL in your own app to use the local model

Prompt for your agent

Using bentoml/OpenLLM, run `openllm serve llama3.2:1b`, then write a Python script with the OpenAI client and base_url 'http://localhost:3000/v1' that streams an answer to <my question>.

Caveat. Gated models such as Llama need a Hugging Face token before they download.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for bentoml/OpenLLM and "Serve Llama 3.2 as an OpenAI compatible API"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer