Aulafy
ESEN
Courses
Courses/Claude Code + Local AI/Quantization and models for coding
Lesson

Quantization and models for coding

For coding with local models, the biggest model doesn't always win. The one that responds well enough, fits with useful context, and can repeat tasks without dragging you into OOM or impossible latency wins.

  • Choose quant based on hardware and task.
  • Measure speed, quality, and context instead of trusting rankings.
  • Prepare a minimal test for local coding agents.

Practical rule

  • Q4: fits on more hardware, good speed, may lose nuance.
  • Q5: common balance for daily use.
  • Q8: more quality, more memory, less margin for context.
  • FP8/KV cache: useful in serving, but stability must be validated.
TerminalCode
# Prueba repetible para comparar modelos
ollama run qwen2.5-coder:7b "Crea tests para una función que parsea fechas ISO."
ollama run qwen2.5-coder:14b "Crea tests para una función que parsea fechas ISO."

# Observa contexto y carga efectiva
ollama ps

# Guarda resultados:
# - tokens/s
# - tiempo al primer token
# - calidad del test
# - si compila
# - memoria usada

Minimal test for a coding model

  • Explain a real bug from your repo.
  • Generate tests that fail before the fix.
  • Propose a small patch.
  • Run lint and build.
  • Compare diff and errors.

Official sources

  • llama.cpp quantization README
  • vLLM quantized KV cache
  • Ollama context length
AI Homelab with Used RTX 3090sLocal code agents with Ollama