- Understand what Q4, Q5, Q8, and the _K_M suffixes mean.
- Choose a quant level based on VRAM, RAM, and use case: chat, RAG, or coding.
- Import a GGUF into Ollama with a Modelfile.
Practical rule
- Q4_K_M: first choice if you're tight on VRAM or want speed.
- Q5_K_M: sweet spot when you can spend a bit more memory for better quality.
- Q8_0: close to high quality, but much heavier; use it if the model fits comfortably.
Mental selection chart
8 GB VRAM: 7B/8B en Q4_K_M 14B solo si aceptas ir justo o bajar contexto 12 GB VRAM: 7B/8B en Q5_K_M o Q8_0 14B en Q4_K_M 16 GB VRAM: 14B en Q5_K_M 30B pequeño en Q4 si el contexto no es enorme 24 GB+ VRAM: 14B en Q8_0 30B/32B en Q4_K_M o Q5_K_M
Import a GGUF into Ollama
Ollama lets you import GGUF models with a Modelfile. Create a folder, save the GGUF, and write:
FROM ./mi-modelo.Q5_K_M.gguf PARAMETER temperature 0.2 PARAMETER num_ctx 8192 SYSTEM """ Eres un asistente técnico preciso. Si no sabes algo, dilo. """
Then create the model:
ollama create mi-modelo-q5 -f Modelfile ollama run mi-modelo-q5
Quantize yourself with llama.cpp
When starting from a model already converted to high-precision GGUF, llama.cpp lets you create a quantized version.
# Ejemplo genérico; el binario puede llamarse llama-quantize o quantize según tu build ./llama-quantize modelo-f16.gguf modelo-Q4_K_M.gguf Q4_K_M ./llama-quantize modelo-f16.gguf modelo-Q5_K_M.gguf Q5_K_M ./llama-quantize modelo-f16.gguf modelo-Q8_0.gguf Q8_0