Quantization and models for coding

• Choose quant based on hardware and task.
• Measure speed, quality, and context instead of trusting rankings.
• Prepare a minimal test for local coding agents.

Practical rule

• Q4: fits on more hardware, good speed, may lose nuance.
• Q5: common balance for daily use.
• Q8: more quality, more memory, less margin for context.
• FP8/KV cache: useful in serving, but stability must be validated.

# Repeatable test to compare models
ollama run qwen2.5-coder:7b "Create tests for a function that parses ISO dates."
ollama run qwen2.5-coder:14b "Create tests for a function that parses ISO dates."

# Watch context and effective load
ollama ps

# Save results:
# - tokens/s
# - time to first token
# - test quality
# - whether it compiles
# - memory used

Minimal test for a coding model

• Explain a real bug from your repo.
• Generate tests that fail before the fix.
• Propose a small patch.
• Run lint and build.
• Compare diff and errors.

Official sources

• llama.cpp quantization README
• vLLM quantized KV cache
• Ollama context length