What do you want to do?
3 use cases
-
Serve Llama 3.2 as an OpenAI compatible API
Run openllm serve on llama3.2:1b and point the OpenAI Python client at localhost:3000/v1.
View prompt
Using bentoml/OpenLLM, run `openllm serve llama3.2:1b`, then write a Python script with the OpenAI client and base_url 'http://localhost:3000/v1' that streams an answer to <my question>.
-
Try open models interactively before you pick one
Use openllm hello and openllm run to chat with models from the terminal and compare them.
View prompt
With bentoml/OpenLLM, run `openllm model list`, pick two small models, chat with each using `openllm run`, and compare their answers to <my three test prompts>.
-
Keep the model catalog current and inspect a model
Update the default model repository and read the details of a model before serving it.
View prompt
Using bentoml/OpenLLM, run `openllm repo update`, then `openllm model get llama3.2:1b`, and summarise what hardware the model needs before serving it.
Quick start with OpenLLM
- Check you have what it needs: Python with pip; A GPU sized for the model you pick; A Hugging Face token for gated models.
- Open bentoml/OpenLLM on GitHub and read the README and license.
- Install with
pip install openllm - Set a Hugging Face token if the model is gated, as the README describes
- Start the server with
openllm serve llama3.2:1b - Create an OpenAI client with base_url http://localhost:3000/v1 and send a chat completion
These steps come from our top use case, Serve Llama 3.2 as an OpenAI compatible API, which also has a copy-paste agent prompt.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for bentoml/OpenLLM
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeAbout this repo
- Who it is forDevelopers who want a self hosted, OpenAI compatible endpoint for an open model.
- Why it is hot13k stars, talked about in 2 AI sources we track
- LanguagePython
- Sourcegithub.com/bentoml/OpenLLM ยท Back to Hot repos
Requirements
- Python with pip
- A GPU sized for the model you pick
- A Hugging Face token for gated models
Add this badge to your README
Maintain OpenLLM? Show readers where to find these use cases.
Markdown
[](https://dedupedai.com/use-cases/bentoml-openllm/?utm_source=github&utm_medium=badge&utm_campaign=bentoml-OpenLLM)
HTML
<a href="https://dedupedai.com/use-cases/bentoml-openllm/?utm_source=github&utm_medium=badge&utm_campaign=bentoml-OpenLLM"><img src="https://dedupedai.com/badge/bentoml/OpenLLM.svg" alt="Use cases on Deduped" height="20"></a>
A static image, no tracking script. About the badge
Questions about OpenLLM
What is OpenLLM?
OpenLLM lets you run open source LLMs such as Llama and Qwen as OpenAI compatible API endpoints with one command, and includes a chat UI and a model repository you can list and update. It is an open-source project on GitHub: https://github.com/bentoml/OpenLLM.
How do I use OpenLLM?
Install with pip install openllm; then set a Hugging Face token if the model is gated, as the README describes; start the server with openllm serve llama3.2:1b. The use cases on this page give the full steps and a copy-paste agent prompt for each job.
Who is OpenLLM for?
Developers who want a self hosted, OpenAI compatible endpoint for an open model.
What can I do with OpenLLM?
3 use cases so far, for example: Serve Llama 3.2 as an OpenAI compatible API; Try open models interactively before you pick one; Keep the model catalog current and inspect a model. Each has steps and a copy-paste agent prompt.
What do I need to run OpenLLM?
Python with pip; A GPU sized for the model you pick; A Hugging Face token for gated models.
What are alternatives to OpenLLM?
Other hot AI repos in Software and DevOps: Skills, Claude Mem, Freellmapi, Andrej Karpathy Skills, Agent Desktop.
Is OpenLLM safe to run?
Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.