LiteLLM as a Gateway and Cost Control

• Understand LiteLLM as a proxy between your app and multiple models.
• Design keys per user, team, or environment.
• Control cost, fallbacks, and cache before surprises happen.

Conceptual setup

model_list:
  - model_name: local-qwen
    litellm_params:
      model: openai/Qwen/Qwen3-8B
      api_base: http://127.0.0.1:8000/v1
      api_key: local

  - model_name: backup-cloud
    litellm_params:
      model: openai/gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY

What to measure per key

• Input and output tokens.
• Estimated or actual cost.
• Model used and fallback applied.
• Latency and errors.
• User, team, or tenant.