Local DeepSeek models and quantization

Three deployment families

For DSH there are three distinct routes: the official DeepSeek API, smaller distilled models, and large quantized models with offloading. Do not turn them into one promise: each one fits a different kind of work.

On 24 GB VRAM, medium distilled models are usually more realistic than trying to run a full large model. Huge Q1/Q2 models with SSD offloading can be useful for batch work, but they are not a fluid agent experience.

• API: best starting point for productivity and long context.
• Local distills: privacy, low variable cost and bounded tasks.
• Q1/Q2 with offload: experimentation and long non-interactive jobs.

DSH decision table

Choose by mission, not ranking. An agent needs stable tool calling, permission following, good repository reading and enough latency for iteration.

If the local model is so slow that you stop reviewing its actions, the system becomes less safe even if it is more private.

Hardware / route           Reasonable DSH use
8-16 GB RAM                small tests with lightweight distills
24 GB VRAM                 14B/32B R1 distills for bounded tasks
Mac 64-128 GB + SSD        slow batch with aggressive quantization
DeepSeek API               daily work, long context and productivity

Minimum benchmark

Do not copy tokens-per-second figures from the internet. Run the same prompt, workspace and mode several times. Record quality and time to a usable result.

Always compare against V4-Flash or V4-Pro so you know whether local savings compensate for speed, quality or coordination loss.

Local vs API DSH measurement:
- exact model and quantization
- runtime used
- hardware and memory
- DSH mode
- configured context
- tokens per second
- cache hit rate
- total time
- accepted result
- retries needed

Connect without making this another tutorial

DSH only needs the local runtime to expose an endpoint compatible with the provider you declare. Installation, GPU, network security and runtime troubleshooting belong in the local AI tutorial.

In this phase, only validate that the endpoint responds, the model name matches and the Trajectory clearly records whether a task ran locally or through the API.

Before using a local model in DSH:
- local runtime already tested outside DSH
- exact model and license reviewed
- endpoint exposed only where intended
- first test uses no sensitive data
- comparison against DeepSeek API saved