Aulafy
ESEN
Courses
Courses/Fine-tuning and post-training for LLMs/Export to GGUF and Ollama
Lesson

Export to GGUF and Ollama

Training is only half the job. If your adapted model cannot be used with your local stack, you are not done yet. Exporting properly means preserving behavior, quantizing thoughtfully, and testing in the real app.

  • Understand when to use a separate adapter and when to merge.
  • Prepare GGUF for llama.cpp/Ollama.
  • Create a Modelfile with a system prompt and parameters.

Output workflow

TerminalCode
adapter LoRA
  -> evaluar
  -> merge con modelo base si procede
  -> export safetensors
  -> convertir a GGUF
  -> cuantizar Q4/Q5/Q8
  -> crear Modelfile Ollama
  -> probar en la app real

Ollama Modelfile

TerminalCode
FROM ./soporte-qwen-lora-q5_k_m.gguf

PARAMETER temperature 0.2
PARAMETER num_ctx 4096

SYSTEM """
Eres un asistente interno de soporte.
Responde en español claro.
Si faltan datos, pide aclaración.
No inventes políticas, precios ni datos personales.
"""
TerminalCode
ollama create soporte-pyme -f Modelfile
ollama run soporte-pyme "Clasifica este email de ejemplo"
Evals, Overfitting, and RegressionsProject: adapted model for an SMB