DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to route one model name across providers with fallback with Respan Gateway

Define a named route that falls back to a second vendor and races a slow call.

Who it is forA backend engineer whose service fails or slows down when one model provider has an outage or latency spike.

The problemA single provider outage or slow response breaks requests or hurts tail latency.

The outcomeClients call one route name, and the gateway switches or hedges to another provider automatically.

What you need first

  • Python with pip, or a Rust toolchain with cargo, or Docker
  • An API key for at least one model vendor, such as OPENAI_API_KEY
  • An existing app or SDK that accepts a custom base URL
  • Optional: a Respan API key for tracing and a tool vendor key for MCP tools

Steps

  1. Create a starter config with respan-gateway init.
  2. Add a [routes.<name>] table whose to list names two providers and whose hedge value sets the delay, following the README example.
  3. Validate the file with respan-gateway check.
  4. Reload the config with --watch or SIGHUP, as the README describes, without restarting the gateway.
  5. Send a request with "model" set to your route name and confirm which provider answered in the live log.

Prompt for your agent

In the respan-gateway repo, edit gateway.toml to add a route that lists two providers in `to` with a hedge delay, then run the check command and send one request using the route name <route-name>. Return which provider served the reply, and verify the check command reports no errors.

Caveat. The README example pairs an Anthropic and an OpenAI model, so each provider needs its own key set before the route can answer.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page