- Set up a reproducible local architecture for an LLM app.
- Document the model, gateway, logs, evals, costs, and limits.
- Leave an exit checklist before opening it to users.
Project architecture
user -> web app -> own API with auth -> LiteLLM gateway -> vLLM or llama.cpp server -> Langfuse/OpenTelemetry -> promptfoo evals -> Redis for queues/cache -> metrics dashboard
Production checklist
Maintenance and drift
An LLM platform degrades when users, documents, prompts, models, or expectations change. Define a simple review before the system fails silently.
drift_check:
frequency: "weekly"
signals:
- rise_in_abstentions
- more_human_edits
- high_p95_latency
- new_question_types
- drop_in_eval_dataset
action:
- review_traces
- update_dataset
- compare_current_model_vs_candidate
- keep_rollback_ready- The model and its hash are documented.
- The server is not exposed directly to the internet.
- There are keys per environment, user, or team.
- There are rate limits and a budget.
- Traces show model, latency, tokens, and errors.
- There are minimal evals before changing prompts or models.
- There is a fallback plan if the local model goes down.
decision_salida: estado: "internal pilot" usuarios: ["support team"] limite_diario_tokens: 500000 modelos: ["local-qwen", "backup-cloud"] datos_permitidos: ["internal manuals", "FAQs"] datos_prohibidos: ["personal contracts", "secrets", "credentials"] siguiente_revision: "2026-07-17"