LiteLLM as a gateway

Mental model

Your application should not need to know whether it calls Ollama, vLLM, OpenAI, Anthropic, or a fallback. It talks to the gateway; the gateway applies policy.

model_list:
  - model_name: local-fast
    litellm_params:
      model: ollama/qwen3:4b
      api_base: http://localhost:11434

  - model_name: frontier-review
    litellm_params:
      model: openai/gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  num_retries: 2
  fallbacks:
    - { local-fast: [frontier-review] }

What to control from the gateway

• Virtual keys per environment, user, or team.
• Budgets and rate limits to avoid surprises.
• Fallbacks by model, tag, or task type.
• Cost, latency, and error logs.

Official sources

• LiteLLM Routing & Load Balancing: https://docs.litellm.ai/docs/routing-load-balancing
• LiteLLM Virtual Keys: https://docs.litellm.ai/docs/proxy/virtual_keys
• LiteLLM Budgets and Rate Limits: https://docs.litellm.ai/docs/proxy/users