Queues, rate limits, and cache

• Separate interactive requests from heavy jobs.
• Apply rate limits per user, team, or tenant.
• Use cache without breaking security or data freshness.

What goes in a queue and what doesn't

• Interactive: chat, autocomplete, actions the user expects on screen.
• Queue: summarize 100 PDFs, generate audio, process videos, evaluate many prompts.
• Scheduled: reindex documents, recalculate embeddings, run nightly evals.

Conceptual rate limits

limites:
  usuario_gratis:
    requests_minuto: 10
    tokens_dia: 20000
  equipo_interno:
    requests_minuto: 60
    tokens_dia: 500000
  trabajos_pesados:
    concurrencia: 2
    reintentos: 1
    timeout_segundos: 300

Responsible caching

• Cache public responses or repeated questions without private data.
• Do not cache responses with personal data without explicit design.
• Include model, prompt version, and permissions in the cache key.
• Invalidate cache when documents or policies change.