# AI use cases for Fastllm Proxy (azrtydxb/Fastllm-proxy)

Repo: [azrtydxb/Fastllm-proxy](https://github.com/azrtydxb/Fastllm-proxy) | Use cases: 3 | Page: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/

fastllm-proxy is a Rust LLM router that puts one OpenAI-compatible endpoint in front of vLLM, SGLang, llama.cpp and 81 providers, with routing rules, RBAC, rate limits, budgets and a built-in management UI.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Put one OpenAI-compatible endpoint in front of every model: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/one-endpoint-for-all-llm-providers/
- Use local GPUs first and spill to the cloud when busy: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/spill-local-gpu-traffic-to-cloud/
- Fall back to a free local model when budget runs low: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/degrade-to-local-model-past-budget/

## How to put one OpenAI-compatible endpoint in front of every model with Fastllm Proxy

Repo: [azrtydxb/Fastllm-proxy](https://github.com/azrtydxb/Fastllm-proxy) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Run the proxy with Docker so apps call one address while it routes to local and cloud backends.

**Who it is for:** A platform engineer tired of every team wiring up provider keys separately.

**The problem:** Each app talking to providers directly means scattered keys and no central control.

**The outcome:** One endpoint on port 4000 and a management UI on 4001.

### Steps

1. Run `git clone https://github.com/azrtydxb/Fastllm-proxy && cd Fastllm-proxy`
2. Start with `docker compose up -d`
3. Set an admin password with the set-password command from the README
4. Point clients at port 4000 and open the UI on 4001

### Prompt to paste into your agent

```text
Deploy azrtydxb/Fastllm-proxy with `docker compose up -d`, add <backend>, and send one chat request through port 4000. Confirm it appears in the management UI.
```

### Caveat

It reads LiteLLM-format config files unchanged, so an existing LiteLLM setup can be pointed at it.

Risk, credentials: The proxy stores provider keys centrally, so protect the management UI and change the example password.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/one-endpoint-for-all-llm-providers/

## How to use local GPUs first and spill to the cloud when busy with Fastllm Proxy

Repo: [azrtydxb/Fastllm-proxy](https://github.com/azrtydxb/Fastllm-proxy) | Difficulty: Advanced | Time: 45 minutes | Basis: From the README

Write first-match routing rules so requests go to your local model until it is full, then to a cloud provider.

**Who it is for:** An ML infra engineer with a few GPUs who wants cloud only for overflow.

**The problem:** Static routing either overloads local GPUs or wastes cloud spend.

**The outcome:** Traffic that stays local while capacity allows and overflows automatically.

### Steps

1. Define a local backend and a cloud provider
2. Add a rule at position 0 with max_inflight_per_backend targeting the local model
3. Add a rule at position 1 targeting the cloud provider
4. Load test and watch where requests land

### Prompt to paste into your agent

```text
Configure fastllm-proxy routing so <local model> takes up to 2 in-flight requests per backend and extra requests go to <cloud provider>. Send 10 parallel requests and report how many hit each target.
```

### Caveat

Rules are evaluated in position order, so a broad rule placed first will shadow later ones.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/spill-local-gpu-traffic-to-cloud/

## How to fall back to a free local model when budget runs low with Fastllm Proxy

Repo: [azrtydxb/Fastllm-proxy](https://github.com/azrtydxb/Fastllm-proxy) | Difficulty: Advanced | Time: 30 minutes | Basis: From the README

Use budget rules so users past 80 percent of budget get the local model instead of an error.

**Who it is for:** A team lead giving developers LLM access under a monthly budget.

**The problem:** Hard budget cutoffs stop work with payment errors.

**The outcome:** Graceful downgrade to a local model once spend crosses a threshold.

### Steps

1. Set budgets per user or team in the UI
2. Add a rule with min_budget_used_percent 80 targeting the local model
3. Label batch work with the x-fastllm-tier header to route it cheaply
4. Watch budget usage in the UI

### Prompt to paste into your agent

```text
In fastllm-proxy, add a rule routing requests to <local model> once min_budget_used_percent reaches 80, then simulate a user over that threshold. Confirm their requests hit the local model instead of returning 402.
```

### Caveat

Budget and RBAC checks run against an in-memory snapshot, so changes apply once the snapshot refreshes.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/degrade-to-local-model-past-budget/