VRAM and OOM: vLLM vs Ollama/llama.cpp

• Separate weight memory, KV cache, context, and batch.
• Decide when to use vLLM, Ollama, or llama.cpp.
• Apply a fix path before buying new hardware.

Quick rule

• Ollama: simpler and more stable for personal use, testing, and low concurrency.
• llama.cpp: lots of control, GGUF, mixed CPU/GPU, and practical multi-GPU.
• vLLM: throughput and concurrency when the model fits with headroom.

# Watch GPU and memory while testing
nvidia-smi -l 1

# Ollama: check model, size, and effective context
ollama ps

# vLLM: start conservatively
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85

Fix path before you give up

• Lower `max-model-len`: long context consumes a lot of memory.
• Try a smaller quant or a smaller model.
• Reduce batch/concurrency.
• Leave VRAM headroom: don't aim for 99% utilization.
• If you need stability, try llama.cpp/Ollama before vLLM.
• If you need concurrency, use vLLM but with a model that fits comfortably.

# llama.cpp server with GGUF model
./llama-server \
  -m ./models/modelo-q4_k_m.gguf \
  --ctx-size 4096 \
  --n-gpu-layers 99 \
  --host 0.0.0.0 \
  --port 8080

Official sources

• vLLM: Quantized KV Cache
• Ollama: context length
• llama.cpp: multi-GPU