DEDUPEDWhat do you want to do?
3 use cases
-
Put one OpenAI-compatible endpoint in front of every model
Run the proxy with Docker so apps call one address while it routes to local and cloud backends.
- REPO
azrtydxb/Fastllm-proxy - Software and DevOps
- Intermediate
View prompt
Deploy azrtydxb/Fastllm-proxy with `docker compose up -d`, add <backend>, and send one chat request through port 4000. Confirm it appears in the management UI.
- REPO
-
Use local GPUs first and spill to the cloud when busy
Write first-match routing rules so requests go to your local model until it is full, then to a cloud provider.
View prompt
Configure fastllm-proxy routing so <local model> takes up to 2 in-flight requests per backend and extra requests go to <cloud provider>. Send 10 parallel requests and report how many hit each target.
-
Fall back to a free local model when budget runs low
Use budget rules so users past 80 percent of budget get the local model instead of an error.
View prompt
In fastllm-proxy, add a rule routing requests to <local model> once min_budget_used_percent reaches 80, then simulate a user over that threshold. Confirm their requests hit the local model instead of returning 402.
Quick start with Fastllm Proxy
- Check you have what it needs: Docker and Docker Compose; At least one inference backend or provider key.
- Open azrtydxb/Fastllm-proxy on GitHub and read the README and license.
- Run
git clone https://github.com/azrtydxb/Fastllm-proxy && cd Fastllm-proxy - Start with
docker compose up -d - Set an admin password with the set-password command from the README
- Point clients at port 4000 and open the UI on 4001
These steps come from our top use case, Put one OpenAI-compatible endpoint in front of every model, which also has a copy-paste agent prompt.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for azrtydxb/Fastllm-proxy
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeFastllm Proxy alternatives · Best AI repos for Software and DevOps
About this repo
- Who it is forPlatform teams serving LLMs to many apps or users who need routing, budgets and access control with very low overhead.
- Why it is hot107 stars, launched 81 days ago
- LanguageRust
- Sourcegithub.com/azrtydxb/Fastllm-proxy · Back to Hot repos
Requirements
- Docker and Docker Compose
- At least one inference backend or provider key
Add this badge to your README
Maintain Fastllm Proxy? Show readers where to find these use cases. 
Markdown
[](https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/?utm_source=github&utm_medium=badge&utm_campaign=azrtydxb-Fastllm-proxy)
HTML
<a href="https://dedupedai.com/use-cases/azrtydxb-fastllm-proxy/?utm_source=github&utm_medium=badge&utm_campaign=azrtydxb-Fastllm-proxy"><img src="https://dedupedai.com/badge/azrtydxb/Fastllm-proxy.svg" alt="Use cases on Deduped" height="20"></a>
A static image, no tracking script. About the badge
Questions about Fastllm Proxy
What is Fastllm Proxy?
fastllm-proxy is a Rust LLM router that puts one OpenAI-compatible endpoint in front of vLLM, SGLang, llama.cpp and 81 providers, with routing rules, RBAC, rate limits, budgets and a built-in management UI. It is an open-source project on GitHub: https://github.com/azrtydxb/Fastllm-proxy.
How do I use Fastllm Proxy?
Run git clone https://github.com/azrtydxb/Fastllm-proxy && cd Fastllm-proxy; then start with docker compose up -d; set an admin password with the set-password command from the README. The use cases on this page give the full steps and a copy-paste agent prompt for each job.
Who is Fastllm Proxy for?
Platform teams serving LLMs to many apps or users who need routing, budgets and access control with very low overhead.
What can I do with Fastllm Proxy?
3 use cases so far, for example: Put one OpenAI-compatible endpoint in front of every model; Use local GPUs first and spill to the cloud when busy; Fall back to a free local model when budget runs low. Each has steps and a copy-paste agent prompt.
What do I need to run Fastllm Proxy?
Docker and Docker Compose; At least one inference backend or provider key.
What are alternatives to Fastllm Proxy?
Other hot AI repos in Software and DevOps: Deep Infer, Harness2, Orca, Opencode, Graphify.
Is Fastllm Proxy safe to run?
Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.