Aulafy
ESEN
Courses
Courses/Local MLOps and model deployment/Serving map: Ollama, llama.cpp, vLLM
Lesson

Serving map: Ollama, llama.cpp, vLLM

Running a model on your machine is the first step. Serving it well means a stable API, limits, logs, metrics, queues, cache, and a clear way to switch models without breaking the app.

  • Choose between Ollama, llama.cpp, vLLM, LiteLLM, and Ray Serve depending on the use case.
  • Distinguish local demos, internal services, and real production.
  • Design a minimal architecture with gateway, observability, and evals.

Quick rule

  • Ollama: local prototypes, teaching, personal apps, and small teams.
  • llama.cpp server: lightweight GGUF, modest CPU/GPU, fine control, and a simple local server.
  • vLLM: GPUs, high performance, batching, OpenAI-compatible API, and large models.
  • LiteLLM: gateway to unify providers, keys, budgets, cache, and fallbacks.
  • Ray Serve: scale multiple replicas, multi-model, autoscaling, and production patterns.

Minimal architecture

TerminalCode
app web
  -> API propia /api/chat
  -> gateway LiteLLM
  -> backend local: llama.cpp | vLLM | Ollama
  -> observabilidad: Langfuse / OpenTelemetry
  -> evals: promptfoo o tests propios
  -> logs: request_id, modelo, latencia, coste, resultado
VRAM and OOM: vLLM vs Ollama/llama.cpp