LoRA and QLoRA without the hype

• Understand what you're actually training with LoRA.
• Choose rank, alpha, learning rate, and epochs wisely.
• Detect early signs of overfitting.

Parameters that matter

• rank r: adapter capacity. More is not always better.
• alpha: scale of LoRA's impact.
• learning rate: learning speed; too high can destroy generalization.
• epochs: passes through the dataset; too many cause memorization.
• target modules: layers where you apply LoRA.

Prudent initial configuration

lora:
  r: 16
  alpha: 32
  dropout: 0.05
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
training:
  learning_rate: 2e-4
  epochs: 1
  max_seq_length: 2048
  eval_steps: 50