Local models vs cloud models: how we choose
· 2 min read
Privacy, cost per token, latency and capability all pull in different directions. Here's the decision rule we actually use.
Every AI feature starts with the same question: where does the model run? There's no universally right answer, but there is a decision procedure.
The two extremes
Cloud models (Anthropic Claude, OpenAI GPT, Google Gemini) run on someone else's GPUs. You send tokens over the wire, you get tokens back, you pay per million.
Local models (Llama, Qwen, DeepSeek, Mistral, Gemma via Ollama, LM Studio, llama.cpp, vLLM) run on hardware you control. You pay for the box, not the tokens.
What actually differs
| Dimension | Cloud | Local |
|---|---|---|
| Capability ceiling | Highest available | One to two generations behind flagship |
| Privacy | Data leaves your network | Nothing leaves the machine |
| Marginal cost | Per token, forever | Zero after hardware |
| Latency | Network + queue | Predictable once loaded |
| Ops burden | None | GPU, memory, updates, monitoring |
| Availability | Vendor outage risk | Yours to keep running |
The decision rule
- Does the data leave the building? If it's PII, medical, legal or client-confidential and there's no signed DPA and a lawful basis, it stays local. This is not a performance trade-off, it's a legal one.
- Is the task verifiable and repetitive? Classification, extraction, embedding, summarisation, routing, tagging — a 7-30B local model does these at production quality. Frontier capability is wasted here.
- Does the task need frontier reasoning? Long-horizon agentic work, complex code generation, subtle judgement calls — cloud wins, and the token cost is usually trivial next to the engineering time.
- Is volume high and the task easy? Local. Paying per token for a job a local model does for free is the most common overspend in AI products.
The pattern we ship
Most systems want both, behind one interface:
request → router → [ local model ] ← cheap, private, bulk
↘ [ cloud model ] ← hard, rare, expensiveThat's a model router — covered separately. The important property: the application code never knows which model answered. It asks for "a completion" and gets one.
Hybrid realities
- Sensitive first, cloud fallback: try local; escalate to cloud only when the local model's confidence or self-check fails, and only for non-sensitive fragments.
- Embeddings local, generation cloud: embeddings are high-volume and low-risk; keep them on your own box.
- Draft local, review cloud: local drafts cheaply; a cloud model critiques and rewrites the top candidates.
What we actually run
Our own stack is local-first with Ollama serving open-weights models for the bulk of agent work, and cloud APIs reserved for the hard tail. The economics are dramatically better, the privacy story is simple, and when we do need frontier quality it's one config line away.
Related posts
- AIHermesAgents
Hermes: an agent that remembers, routes and ships
Skills, persistent memory, model routing and MCP tooling — what an agent runtime needs before it's useful for real work.
· 2 min read
- EngineeringModel RoutingLLM
Model routers: one endpoint, many models
A router picks the right model per request — by task, cost, latency or availability. It's the single highest-leverage piece of AI infrastructure you can build.
· 2 min read