Project: private mini LLM platform

• Set up a reproducible local architecture for an LLM app.
• Document the model, gateway, logs, evals, costs, and limits.
• Leave an exit checklist before opening it to users.

Project architecture

usuario
  -> app web
  -> API propia con auth
  -> LiteLLM gateway
  -> vLLM o llama.cpp server
  -> Langfuse/OpenTelemetry
  -> promptfoo evals
  -> Redis para colas/caché
  -> dashboard de métricas

Production checklist

• The model and its hash are documented.
• The server is not exposed directly to the internet.
• There are keys per environment, user, or team.
• There are rate limits and a budget.
• Traces show model, latency, tokens, and errors.
• There are minimal evals before changing prompts or models.
• There is a fallback plan if the local model goes down.

decision_salida:
  estado: "piloto interno"
  usuarios: ["equipo soporte"]
  limite_diario_tokens: 500000
  modelos: ["local-qwen", "backup-cloud"]
  datos_permitidos: ["manuales internos", "FAQs"]
  datos_prohibidos: ["contratos personales", "secretos", "credenciales"]
  siguiente_revision: "2026-07-17"