II · THE IDEA · ARTIFICIAL INTELLIGENCE
Local Versus Cloud, Honestly
▶ Listen · narrated
The question is not whether local inference is possible — it demonstrably is — but whether the five trade-offs involved align with what you actually need from the system today.
At a glance
- Local
- Model runs on your hardware; no data leaves your machine
- Cloud
- Model runs on provider's infrastructure; you send requests over the network
- Key axes
- Privacy, cost structure, latency, capability ceiling, maintenance burden
- Tooling
- Ollama and llama.cpp enable local inference on consumer hardware
Think of cloud inference like renting a car: you pay each time you use it, someone else handles maintenance, and you can rent a better car than you could afford to own. Local inference is like owning a car: you pay upfront, then each trip is nearly free, but you handle all the upkeep yourself and you are limited to what you can afford to buy. Neither is wrong. Renting makes sense if you drive occasionally or need a luxury vehicle sometimes. Owning makes sense if you drive every day and want control over everything. The same logic applies here. Cloud APIs charge per use, stay up to date automatically, and give you access to the most capable models. Local inference requires you to buy hardware and manage software, but then your data never leaves your machine and heavy use costs almost nothing. The best choice depends on how often you use it, how much privacy matters, and whether you need the absolute best performance or just something good enough.
Local inference using tools like llama.cpp and Ollama runs quantised models on consumer GPUs or unified-memory systems, typically using 4-bit or 8-bit quantisation to fit larger models into limited VRAM. A 24 GB GPU can run a quantised 70-billion-parameter model at roughly 5-15 tokens per second, depending on context length and batch size. Cloud APIs run full-precision or lightly quantised models on A100 or H100 clusters, delivering 30-100+ tokens per second with lower time-to-first-token due to optimised serving infrastructure. Privacy is structural: local inference keeps data on-device, while cloud inference requires transmitting prompts and responses over TLS to the provider's infrastructure, where logging and retention policies vary. Cost structures differ fundamentally: cloud APIs charge per million tokens (typically $0.50-$20 depending on model size), scaling linearly with usage, while local inference has a fixed capital cost (GPU hardware, typically $800-$3000 for capable consumer cards) plus negligible marginal cost per inference. The capability gap is measurable: frontier cloud models outperform open-weight models on benchmarks like MMLU, GSM8K, and HumanEval by 5-15 percentage points, though the gap narrows for narrower tasks and fine-tuned models. Maintenance overhead for local setups includes model downloads (tens of GB per model), dependency management (CUDA drivers, Metal support, or CPU-only builds), and version compatibility between llama.cpp, Ollama, and model formats. Latency for local inference eliminates network overhead (typically 50-200ms for cloud) but suffers from slower per-token generation on consumer hardware, making the total time competitive only for short outputs or when time-to-first-token dominates.
Look closer
Privacy is binary, but the threat model varies
Local inference means your prompts and the model's responses never traverse a network. No logs, no terms of service, no question of what the provider does with your data. That matters enormously if you are drafting legal documents, processing patient records, or simply uncomfortable with surveillance. But privacy is not the same as security: a compromised laptop leaks just as badly as a compromised API, and local models still embed the biases and memorised fragments of their training data. The privacy gain is real, but it is about transmission and third-party access, not about the model being inherently safer.
Cost structures differ in kind, not just degree
Cloud APIs charge per token, so cost scales directly with usage. A prototype costs pennies, a production service with millions of requests costs thousands, and you pay nothing when idle. Local inference inverts this: you pay the hardware cost upfront — whether that is a gaming GPU you already own or a machine you buy specifically for this — and then inference is free at the margin, constrained only by electricity and time. For intermittent use, cloud is cheaper. For sustained high-volume work, local can amortise quickly. But the comparison depends entirely on your usage pattern, and many people misjudge their own.
Capability is not fixed, but it is real
The largest and most capable models do not fit on consumer hardware, even with quantisation. A high-end GPU with 24 GB of memory can run a quantised 70-billion-parameter model slowly, but not a 405-billion-parameter one, and certainly not the frontier models that cloud providers deploy on clusters. Local inference gives you access to open-weight models, which are improving rapidly, but they lag the state of the art by a margin that matters for difficult tasks. The gap is narrowing, but it has not closed, and pretending otherwise does not help anyone make an informed choice.
The story
The choice between local and cloud inference is not ideological. It is a bundle of five distinct trade-offs, and the right answer depends on what you are doing and what you value.
Privacy is the most straightforward axis. If you run a model locally, your data never leaves your machine. No API provider sees your prompts, no logs accumulate on a distant server, no terms of service govern what happens to your queries. For anyone working with confidential material — legal drafts, medical records, proprietary code — this is not a minor convenience. It is a requirement. Cloud providers offer assurances, but assurances are not the same as impossibility, and local inference makes interception structurally impossible because there is no transmission.
Cost is less obvious because the structures differ. Cloud APIs charge per token generated, so a single experiment costs almost nothing and a high-traffic application costs thousands of dollars a month. You pay for what you use, and you pay nothing when idle. Local inference inverts this entirely. You pay upfront for the hardware — a GPU, or a machine with enough unified memory — and after that, inference is free at the margin. Electricity costs something, but not much. For occasional use, cloud is clearly cheaper. For sustained high-volume work, local hardware can pay for itself in months. The break-even point depends on your usage, and most people guess wrong about their own patterns until they measure them.
Latency splits two ways. Cloud inference depends on network round-trips, so you pay tens or hundreds of milliseconds before the first token even starts generating. Local inference skips that entirely: the model is already resident in memory, and generation starts immediately. For interactive applications, that difference is perceptible. But cloud providers run models on expensive accelerators optimised for throughput, so once generation starts, tokens arrive quickly. Local inference on consumer hardware is slower per token, sometimes much slower. A quantised model on a laptop GPU might generate five tokens per second where a cloud API generates fifty. The winner depends on whether you care more about time-to-first-token or total time-to-completion.
Capability is the axis where ideology most often clouds the facts. The largest and most capable models do not fit on consumer hardware. A high-end GPU with 24 GB of memory can run a quantised 70-billion-parameter model, but not comfortably, and not a 405-billion-parameter model at all. Frontier cloud models run on clusters with hundreds of gigabytes of memory, and they are measurably better at difficult tasks. Open-weight models are improving quickly, and for many tasks they are now good enough that the capability gap does not matter. But the gap exists, and pretending it does not helps no one. If you need the best performance on complex reasoning or extended context, cloud is currently the only option.
Maintenance is the final axis, and it is easy to underestimate. Cloud APIs abstract everything: you send a request, you get a response, and someone else worries about uptime, updates, and driver compatibility. Local inference means you install the software, manage dependencies, download multi-gigabyte model files, and troubleshoot when something breaks. Tools like Ollama and llama.cpp have made this vastly easier than it was even a year ago, but it is still more work than calling an API. For developers comfortable with command-line tools, the burden is light. For everyone else, it is a real cost.
None of these axes reduces to the others. You cannot optimise for all five simultaneously, and the correct trade-off depends on your situation. A lawyer drafting a sensitive contract has different needs than a startup building a chatbot, and both have different needs than a researcher experimenting with prompts. The choice is not between good and bad. It is between different bundles of constraints, and the honest answer is that both options are correct for different people.
Why it mattered then
Local inference became practical for non-specialists only recently. The release of llama.cpp in 2023 provided a fast, portable implementation that could run quantised models on consumer hardware without requiring CUDA or specialised drivers. Ollama followed shortly after, wrapping llama.cpp in a simple interface that handled model downloads, memory management, and API serving with a few commands. Before these tools, running a large language model locally required either expensive hardware or deep technical knowledge of ML frameworks, GPU drivers, and memory optimisation. The shift mattered because it turned local inference from a specialist capability into something anyone with a decent laptop could attempt. Open-weight models like Llama 2 appeared at the same time, giving people something capable to run. The combination of accessible tooling and accessible models created a genuine alternative to cloud APIs for the first time, and the choice became a real one rather than a theoretical one.
Why it matters now
The trade-offs are shifting quickly, which is why the choice matters more now than it did even six months ago. Open-weight models are improving fast enough that the capability gap with frontier cloud models is narrowing for many tasks, though it has not closed. Quantisation techniques are better, so larger models fit on smaller hardware. Tools like Ollama have eliminated most of the installation friction. At the same time, cloud APIs are cheaper and faster than they were a year ago, and the largest models available through APIs have no local equivalent. The result is that neither option dominates, and the correct choice depends on specifics: what you are building, what you are willing to pay, what your privacy requirements are, and how much capability you actually need. The calculus will shift again as models improve and hardware gets cheaper, but right now the decision is finely balanced, and worth thinking through carefully rather than assuming one answer fits everyone.
The surprising detail
The privacy advantage of local inference is real but narrower than it appears. Running a model locally means your prompts never leave your machine, which eliminates a large class of privacy risks. But the model itself was trained on data scraped from the internet, and it has memorised fragments of that data. If you prompt it carefully, it may reproduce training examples nearly verbatim, which means private information from someone else's documents could appear in your output. Local inference protects your input, but it does not guarantee that the output is free of other people's private information. The model is not a neutral tool; it is a compressed, imperfect copy of its training set, and that set contained things its creators did not intend to include.
Remember this
Neither local nor cloud inference is obviously correct. The choice depends on five separate axes, and optimising for one often means compromising on another.
Test yourself
You are building a tool that generates summaries of medical records. The records are highly sensitive, and the tool will be used several thousand times a day. Which deployment choice is more defensible, and why?
Local inference is more defensible, despite the higher upfront cost and maintenance burden. Medical records are subject to strict privacy regulations, and sending them to a cloud API — even one with strong contractual assurances — introduces legal and ethical risks that local inference avoids entirely. The high usage volume means the hardware cost will amortise quickly, likely within months, so the cost argument favours local deployment as well. The capability gap matters less here because summarisation is a task where current open-weight models perform well enough for production use. Latency and maintenance are real costs, but they are manageable, and neither outweighs the privacy and regulatory requirements.
Go deeper
- FAQ - Ollama · docs.ollama.com
- llama.cpp/README.md at master · ggml-org/llama.cpp · GitHub · github.com
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.