Observability with Langfuse and OpenTelemetry

• Record useful traces without leaking sensitive data.
• Measure latency, errors, cost, and quality per route.
• Use request_id to follow a complete request.

What to log

• Technical identity: request_id, route, app version.
• Model: provider, name, version, and parameters.
• Performance: total latency, tokens, errors, and retries.
• RAG: retrieved documents, not necessarily full text.
• Quality: associated eval, feedback, and expected outcome.

Minimum event

Experiment tracking and lightweight registry

Besides per-request traces, store which model, prompt, dataset, and configuration version produced each result. You can start with a simple table before setting up a full registry.

model_registry:
  id: "support-router-v3"
  base_model: "qwen3:8b-q5"
  runtime: "ollama"
  prompt_version: "support-en-v7"
  eval_dataset: "support-evals-2026-07"
  score:
    answer_ok: 0.86
    refusal_ok: 0.94
    citation_ok: 0.81
  status: "pilot"
  owner: "support-team"

MLflow, Langfuse, OpenTelemetry, or your own table can coexist. The important part is not losing the relationship between version, eval, deployment, and observed result.

{
  "request_id": "req_20260703_001",
  "route": "/api/chat",
  "model": "local-qwen",
  "prompt_version": "support-v3",
  "latency_ms": 1840,
  "input_tokens": 620,
  "output_tokens": 210,
  "retrieved_docs": ["manual-001#p4", "faq-009#p1"],
  "error": null
}