Aulafy
ESEN
Courses
Courses/Local MLOps and model deployment/Migrate a cloud agent to local
Lesson

Migrate a cloud agent to local

Migrating to local isn't just changing a URL and calling it done. Latency, context, tool calling, quality, observability, and hardware limits all change. Do it in stages and keep an escape hatch.

  • Inventory cloud dependencies before migrating.
  • Test compatibility of prompts, tools, and context.
  • Decide what stays local and what keeps an external fallback.
TerminalCode
migration_matrix:
  tasks:
    classify_email:
      target: "local"
      model: "ollama/qwen2.5:7b"
      risk: "low"
    draft_legal_reply:
      target: "hybrid"
      local_first: true
      cloud_requires_approval: true
    edit_codebase:
      target: "local_with_review"
      requires_tests: true

Tests before changing production

  • Same input against cloud and local.
  • Compare quality, latency, and format.
  • Test minimal tool calling.
  • Measure real and truncated context.
  • Record failures that require fallback.

Official sources

  • LiteLLM routing and load balancing
  • LiteLLM reliability and fallbacks
  • Ollama API
  • vLLM OpenAI-compatible server
Hybrid local/cloud routing with LiteLLMObservability with Langfuse and OpenTelemetry