DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to run a shared self-hosted AI Gateway for your team with Litellm

Deploy one proxy that gives your team OpenAI-style access to many models with virtual keys.

Who it is forA platform or IT admin who needs to give several developers access to LLM providers without sharing raw provider keys.

The problemSharing provider keys across a team makes spend and access hard to control, and each app needs its own provider setup.

The outcomeTeam members send requests to one local endpoint in OpenAI format, using keys the gateway issues.

What you need first

  • Python and the uv package manager installed
  • API keys for the LLM providers you plan to call, such as OpenAI or Anthropic
  • For the gateway use cases: a machine to run the proxy server on and a LiteLLM master key or virtual key for clients

Steps

  1. Install the proxy with uv tool install 'litellm[proxy]'
  2. Start the gateway with litellm --model gpt-4o
  3. Follow the E2E tutorial at https://docs.litellm.ai/docs/proxy/docker_quick_start to create virtual keys
  4. Point a client at http://0.0.0.0:4000 using the OpenAI client and a virtual key as api_key

Prompt for your agent

In the litellm repo at <path>, start the AI Gateway proxy on port 4000 and send a test chat completion for model gpt-4o from a Python OpenAI client pointed at http://0.0.0.0:4000. Expected output: a valid chat response. Verify the proxy log shows the request was received.

Caveat. The README's quick start runs a local endpoint on 0.0.0.0:4000, so you must add your own network, TLS and access controls before exposing it to a team.

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer