# AI use cases for OpenLLM (bentoml/OpenLLM)

Repo: [bentoml/OpenLLM](https://github.com/bentoml/OpenLLM) | Use cases: 3 | Page: https://dedupedai.com/use-cases/bentoml-openllm/

OpenLLM lets you run open source LLMs such as Llama and Qwen as OpenAI compatible API endpoints with one command, and includes a chat UI and a model repository you can list and update.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Serve Llama 3.2 as an OpenAI compatible API: https://dedupedai.com/use-cases/bentoml-openllm/serve-llama-3-2-as-an-openai-compatible-api-with-openllm/
- Try open models interactively before you pick one: https://dedupedai.com/use-cases/bentoml-openllm/try-open-models-interactively-with-openllm-hello/
- Keep the model catalog current and inspect a model: https://dedupedai.com/use-cases/bentoml-openllm/keep-the-openllm-model-catalog-up-to-date/

## How to serve Llama 3.2 as an OpenAI compatible API with OpenLLM

Repo: [bentoml/OpenLLM](https://github.com/bentoml/OpenLLM) | Difficulty: Beginner | Time: 20 minutes | Basis: From the README

Run openllm serve on llama3.2:1b and point the OpenAI Python client at localhost:3000/v1.

**Who it is for:** A developer whose app already uses the OpenAI client and wants a self hosted model.

**The problem:** Switching an app to a local model usually means rewriting its API calls.

**The outcome:** A local server at http://localhost:3000/v1 that the OpenAI client can call unchanged.

### Steps

1. Install with `pip install openllm`
2. Set a Hugging Face token if the model is gated, as the README describes
3. Start the server with `openllm serve llama3.2:1b`
4. Create an OpenAI client with base_url http://localhost:3000/v1 and send a chat completion
5. Swap the base URL in your own app to use the local model

### Prompt to paste into your agent

```text
Using bentoml/OpenLLM, run `openllm serve llama3.2:1b`, then write a Python script with the OpenAI client and base_url 'http://localhost:3000/v1' that streams an answer to <my question>.
```

### Caveat

Gated models such as Llama need a Hugging Face token before they download.

Risk, download: Model weights download on first serve.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/bentoml-openllm/serve-llama-3-2-as-an-openai-compatible-api-with-openllm/

## How to try open models interactively before you pick one with OpenLLM

Repo: [bentoml/OpenLLM](https://github.com/bentoml/OpenLLM) | Difficulty: Beginner | Time: 30 minutes | Basis: From the README

Use openllm hello and openllm run to chat with models from the terminal and compare them.

**Who it is for:** Anyone choosing which open model to use for a project.

**The problem:** Comparing models usually means setting up each one separately.

**The outcome:** A quick hands on comparison of a few models from one tool.

### Steps

1. Install with `pip install openllm`
2. Run `openllm hello` for the guided start
3. List models with `openllm model list`
4. Chat with one model using `openllm run llama3:8b`
5. Note which model answers your sample prompts best

### Prompt to paste into your agent

```text
With bentoml/OpenLLM, run `openllm model list`, pick two small models, chat with each using `openllm run`, and compare their answers to <my three test prompts>.
```

### Caveat

Larger models need a GPU with enough memory; start with a small one.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/bentoml-openllm/try-open-models-interactively-with-openllm-hello/

## How to keep the model catalog current and inspect a model with OpenLLM

Repo: [bentoml/OpenLLM](https://github.com/bentoml/OpenLLM) | Difficulty: Beginner | Time: 10 minutes | Basis: From the README

Update the default model repository and read the details of a model before serving it.

**Who it is for:** A team running OpenLLM that wants the newest model entries.

**The problem:** New open models appear often and the local catalog goes stale.

**The outcome:** An updated catalog and the details of the model you plan to deploy.

### Steps

1. Install with `pip install openllm`
2. Run `openllm repo update` to refresh the model repository
3. Run `openllm model get llama3.2:1b` to see a model's details
4. Serve the model you chose with `openllm serve llama3.2:1b`

### Prompt to paste into your agent

```text
Using bentoml/OpenLLM, run `openllm repo update`, then `openllm model get llama3.2:1b`, and summarise what hardware the model needs before serving it.
```

### Caveat

Gated models such as Llama need a Hugging Face token before they download.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/bentoml-openllm/keep-the-openllm-model-catalog-up-to-date/
