- Understand what Q4, Q5, Q8, and the _K_M suffixes mean.
- Choose a quant level based on VRAM, RAM, and use case: chat, RAG, or coding.
- Import a GGUF into Ollama with a Modelfile.
Practical rule
- Q4_K_M: first choice if you're tight on VRAM or want speed.
- Q5_K_M: sweet spot when you can spend a bit more memory for better quality.
- Q8_0: close to high quality, but much heavier; use it if the model fits comfortably.
Mental selection chart
8 GB VRAM: 7B/8B in Q4_K_M 14B only if you're okay running tight or reducing context 12 GB VRAM: 7B/8B in Q5_K_M or Q8_0 14B in Q4_K_M 16 GB VRAM: 14B in Q5_K_M 30B small in Q4 if context isn't huge 24 GB+ VRAM: 14B in Q8_0 30B/32B in Q4_K_M or Q5_K_M
Import a GGUF into Ollama
Ollama lets you import GGUF models with a Modelfile. Create a folder, save the GGUF, and write:
FROM ./mi-modelo.Q5_K_M.gguf PARAMETER temperature 0.2 PARAMETER num_ctx 8192 SYSTEM """ You are a precise technical assistant. If you don't know something, say so. """
Then create the model:
ollama create mi-modelo-q5 -f Modelfile ollama run mi-modelo-q5
Quantize yourself with llama.cpp
When starting from a model already converted to high-precision GGUF, llama.cpp lets you create a quantized version.
# Generic example; the binary may be called llama-quantize or quantize depending on your build ./llama-quantize modelo-f16.gguf modelo-Q4_K_M.gguf Q4_K_M ./llama-quantize modelo-f16.gguf modelo-Q5_K_M.gguf Q5_K_M ./llama-quantize modelo-f16.gguf modelo-Q8_0.gguf Q8_0