Aulafy
ESEN
Courses
Courses/Local MLOps and model deployment/VRAM and OOM: vLLM vs Ollama/llama.cpp
Lesson

VRAM and OOM: vLLM vs Ollama/llama.cpp

Most local setups don't fail because the model is 'bad': they fail because it doesn't fit, because context blows up the KV cache, or because you're using a runtime built for concurrency when all you need is stability.

  • Separate weight memory, KV cache, context, and batch.
  • Decide when to use vLLM, Ollama, or llama.cpp.
  • Apply a fix path before buying new hardware.

Quick rule

  • Ollama: simpler and more stable for personal use, testing, and low concurrency.
  • llama.cpp: lots of control, GGUF, mixed CPU/GPU, and practical multi-GPU.
  • vLLM: throughput and concurrency when the model fits with headroom.
TerminalCode
# Observa GPU y memoria mientras pruebas
nvidia-smi -l 1

# Ollama: mira modelo, tamaño y contexto efectivo
ollama ps

# vLLM: empieza conservador
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85

Fix path before you give up

  • Lower `max-model-len`: long context consumes a lot of memory.
  • Try a smaller quant or a smaller model.
  • Reduce batch/concurrency.
  • Leave VRAM headroom: don't aim for 99% utilization.
  • If you need stability, try llama.cpp/Ollama before vLLM.
  • If you need concurrency, use vLLM but with a model that fits comfortably.
TerminalCode
# llama.cpp server con modelo GGUF
./llama-server \
  -m ./models/modelo-q4_k_m.gguf \
  --ctx-size 4096 \
  --n-gpu-layers 99 \
  --host 0.0.0.0 \
  --port 8080

Official sources

  • vLLM: Quantized KV Cache
  • Ollama: context length
  • llama.cpp: multi-GPU
Serving map: Ollama, llama.cpp, vLLMllama.cpp server locally