Serving map: Ollama, llama.cpp, vLLM

• Choose between Ollama, llama.cpp, vLLM, LiteLLM, and Ray Serve depending on the use case.
• Distinguish local demos, internal services, and real production.
• Design a minimal architecture with gateway, observability, and evals.

Quick rule

• Ollama: local prototypes, teaching, personal apps, and small teams.
• llama.cpp server: lightweight GGUF, modest CPU/GPU, fine control, and a simple local server.
• vLLM: GPUs, high performance, batching, OpenAI-compatible API, and large models.
• LiteLLM: gateway to unify providers, keys, budgets, cache, and fallbacks.
• Ray Serve: scale multiple replicas, multi-model, autoscaling, and production patterns.

Minimal architecture

web app
  -> own API /api/chat
  -> LiteLLM gateway
  -> local backend: llama.cpp | vLLM | Ollama
  -> observability: Langfuse / OpenTelemetry
  -> evals: promptfoo or own tests
  -> logs: request_id, model, latency, cost, result