What you will learn
- Install and verify Ollama on Windows, macOS, or Linux.
- Run a first local model, inspect the processor and test the local API.
- Understand context, GPU offloading, local versus cloud models, and the API.
- Know when Ollama is the model runtime and when an agent is doing the work.
Ollama is a runtime, not a model or agent
Ollama downloads models, loads them into memory, runs inference, manages context, and exposes an API. Qwen, Gemma, Llama, Mistral, and gpt-oss are models that can run through it. A coding agent such as Codex or Claude Code is another layer: it gives a model tools, permissions, and an action loop.
Your application
↓
Ollama
↓
AI model
↓
CPU / GPU / unified memoryThat distinction matters for safety. Ollama does not edit a repository or send an email by itself. The surrounding application or agent decides whether a tool can do that.
Is Ollama really local?
A downloaded local model can run on your machine and your application normally talks to http://localhost:11434. Ollama also supports cloud models, which are offloaded to Ollama's service and require an account. Do not treat the Ollama app itself as proof that inference is local: check the model and the endpoint before sending private data.
Choose realistic hardware expectations
There is no universal RAM number for a model. Quantization, model architecture, context length, KV cache, multimodal input, and GPU offloading all change memory use. Treat these as starting points, then measure on your own machine.
- 8 GB RAM: small 1B-4B models, basic chat and experimentation.
- 16 GB RAM: many 3B-8B quantized models and light local apps.
- 32 GB RAM: more comfortable document, RAG, and longer-context experiments.
- 64 GB or more: larger models, heavier contexts, agents, or several local services.
Choose hardware and models that really fit your machine
Install Ollama
Download the official application for Windows or macOS. On macOS, Ollama requires Sonoma (14) or newer; Apple M series has CPU and GPU support while Intel Macs are CPU-only. On Linux, use the official installer and verify the service. Keep port 11434 on localhost until you have authentication and network controls in place.
# Linux curl -fsSL https://ollama.com/install.sh | sh # All systems ollama --version ollama list # Linux service check systemctl status ollama
Run your first model
Start with a small model available in the official library. The exact tags change, so copy the current tag rather than treating a tutorial example as permanent. Qwen3 4B is a reasonable small-model example when its current library tag fits your hardware.
ollama run qwen3:4b # Inside the chat: Explain local AI in two sentences. # Leave the chat: /bye
The commands you will actually use
ollama ollama list ollama ps ollama run qwen3:4b ollama pull qwen3:4b ollama rm MODEL:TAG
ollama list shows stored models. ollama ps shows loaded models, their processor split, allocated context, and unload time. Run it before debugging a slow application.
Verify the local API before building anything
Ollama's local API is served by default at http://localhost:11434/api. First check the service, then make one non-streaming request. If these fail, fix Ollama before debugging RAG, Docker, or an agent.
curl http://127.0.0.1:11434/api/tags
curl http://127.0.0.1:11434/api/generate -d '{
"model": "qwen3:4b",
"prompt": "Reply with exactly: OLLAMA API WORKS",
"stream": false
}'Check CPU, GPU, and context
A successful response does not prove GPU acceleration. Run a model, then inspect ollama ps. On NVIDIA systems, nvidia-smi can provide a second signal while generation is running. Do not assume a GPU backend works from its name alone: driver, operating system, model size, and available VRAM all matter.
ollama ps # NVIDIA, while a model is generating nvidia-smi
Ollama is not using the GPU on Windows: structured diagnosis
Context length: use enough, not the maximum
Ollama documents defaults of 4K below 24 GiB VRAM, 32K at 24-48 GiB, and 256K at 48 GiB or more. It recommends at least 64K for web search, coding tools, and agents. More context needs more memory, so increasing it blindly can make a usable setup slow or force CPU offloading.
# One documented way to set a 64K context when serving Ollama OLLAMA_CONTEXT_LENGTH=64000 ollama serve # Check the effective context and processor split ollama ps
Ollama context length: what 4K, 32K, and 64K change
Use the native API or OpenAI-compatible API
The native API is the most direct option for a local app. Ollama also supports parts of the OpenAI API, so an existing application can often use http://localhost:11434/v1/ as its base URL. Compatibility is endpoint- and field-specific, not a promise that every OpenAI feature behaves identically.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1/",
api_key="ollama", # required by the client, ignored locally
)
response = client.chat.completions.create(
model="qwen3:4b",
messages=[{"role": "user", "content": "What does Ollama do?"}],
)
print(response.choices[0].message.content)Embeddings, RAG, vision, and tools come next
Ollama can serve embeddings for semantic search and RAG, and compatible models can support vision or tool calling. Those are separate design problems from getting a first chat running: an embedding model, retrieval quality, citations, permissions, and data retention all need their own evaluation.
Build private document search with Open WebUI, Ollama, and Qdrant
Tool calling with local models
Codex and Claude Code: Ollama runs the model, the agent runs the workflow
Current Ollama integrations can launch supported coding tools. Codex and Claude Code need a model with adequate context, tool-call quality, speed, and memory; a small model that feels good in chat may fail on a repository. Start in a disposable project and keep normal permission and review controls.
ollama launch codex ollama launch claude # Inspect available integrations on your installed version ollama
Connect Claude Code to local AI
Build coding agents with local models
Common mistakes
- Installing a model that is too large, then blaming the prompt instead of memory or CPU offloading.
- Assuming the GPU is active without checking ollama ps or a hardware monitor.
- Raising context to 64K or 256K without measuring the additional memory cost.
- Exposing localhost:11434 to a network or the public internet without authentication and access controls.
- Treating a cloud model as local because it is launched from the Ollama CLI.
- Confusing a model runtime with an agent that can call tools or edit files.
Your practical mission
Save one short diagnostic note: operating system, Ollama version, model tag, output of ollama ps, and whether the API check passed. That is the evidence you need for the next lesson and the first thing to share when asking for help.
ollama --version ollama run qwen3:4b ollama ps curl http://127.0.0.1:11434/api/tags
Official sources, checked September 4, 2026
Ollama API introduction and OpenAI compatibility