Pattern
primary = call_model("stable", prompt)
shadow = call_model("candidate", prompt)
log_comparison({
"prompt_id": prompt.id,
"primary_model": "stable",
"shadow_model": "candidate",
"primary_output": primary.text,
"shadow_output": shadow.text,
"shown_to_user": "primary"
})What to compare
- Perceived quality and human edit rate.
- Cost and latency.
- Format, citation, or safety failures.
- Regressions by task type.
Do not switch the winning model because of one anecdote. Switch when you have enough cases per category and understand what improves and what gets worse.