DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to use local GPUs first and spill to the cloud when busy with Fastllm Proxy

Write first-match routing rules so requests go to your local model until it is full, then to a cloud provider.

Short answer: Define a local backend and a cloud provider; then add a rule at position 0 with max_inflight_per_backend targeting the local model; add a rule at position 1 targeting the cloud provider. All steps.

Who it is forAn ML infra engineer with a few GPUs who wants cloud only for overflow.

The problemStatic routing either overloads local GPUs or wastes cloud spend.

The outcomeTraffic that stays local while capacity allows and overflows automatically.

What you need first

  • Docker and Docker Compose
  • At least one inference backend or provider key

Steps

  1. Define a local backend and a cloud provider
  2. Add a rule at position 0 with max_inflight_per_backend targeting the local model
  3. Add a rule at position 1 targeting the cloud provider
  4. Load test and watch where requests land

Prompt for your agent

Configure fastllm-proxy routing so <local model> takes up to 2 in-flight requests per backend and extra requests go to <cloud provider>. Send 10 parallel requests and report how many hit each target.

Caveat. Rules are evaluated in position order, so a broad rule placed first will shadow later ones.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for azrtydxb/Fastllm-proxy and "Use local GPUs first and spill to the cloud when busy"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer