II · THE IDEA · ARTIFICIAL INTELLIGENCE
Running an Open-Weight Model Locally
▶ Listen · narrated
An open-weight model is numbers in a file. Getting from that file to a working conversation requires a runtime, which determines your speed, memory use, and what you can tune.
At a glance
- llama.cpp
- C++ inference engine, cross-platform, fine control over quantisation and sampling
- Ollama
- Wraps llama.cpp in a simple CLI and REST API, manages models like Docker images
- MLX
- Apple's NumPy-like framework with unified memory, optimised for Metal on M-series chips
- Common input
- All three can load models in GGUF format or convert from Hugging Face checkpoints
Think of an open-weight model as a recipe written in a language your kitchen does not speak. The recipe is public and detailed—every ingredient, every measurement—but you still need a translator and a set of pans that fit your stove. llama.cpp, Ollama and MLX are three different translators, each designed for a different kind of kitchen. llama.cpp is the translator that shows you every step and lets you adjust the heat on every burner. Ollama is the translator that reads the recipe, sets the oven temperature, and tells you when the dish is ready, hiding almost all the intermediate steps. MLX is the translator that only works in kitchens with a specific brand of induction hob, but in those kitchens it is faster and uses less energy because it knows exactly how that hob behaves. The recipe is the same in all three cases. The difference is how much control you have over the cooking process and which stove you are using.
All three tools load a set of model weights, build a computation graph for the transformer architecture, allocate memory for the key-value cache, and execute a forward pass for each token. llama.cpp is written in C++ and compiles to a single binary with optional GPU acceleration via CUDA, Metal, Vulkan or OpenCL. It memory-maps GGUF files directly, which means weights are paged into RAM on demand rather than loaded all at once, and it supports quantisation formats from 2-bit to 8-bit per weight. The command-line interface exposes flags for context length, batch size, thread count, layer offloading and sampling parameters, and the output includes token-level timings and memory statistics. Ollama wraps llama.cpp in a daemon that manages model downloads, caching and lifecycle, and exposes a REST API on localhost. It chooses default inference parameters based on the model and available hardware, and it handles concurrent requests by loading the model once and sharing the KV cache where possible. MLX is a Python framework that compiles operations to Metal Performance Shaders and treats unified memory as a single address space, so models larger than GPU VRAM can still run without explicit host-device transfers. The API is designed to resemble NumPy and PyTorch, and the quantisation formats are specific to MLX rather than compatible with GGUF. All three can load models from Hugging Face checkpoints, but the conversion process differs: llama.cpp uses a Python script that outputs GGUF, Ollama imports GGUF files via a Modelfile, and MLX uses `mlx.core.save` to write its own format.
Look closer
Ollama hides almost all the knobs
Install Ollama, type `ollama run llama3.2`, and you have a working chat session with no configuration file and no Python environment. The model downloads automatically, quantised weights are cached locally, and the interface looks like any other command-line chat. Behind that simplicity, Ollama is calling llama.cpp and choosing reasonable defaults for context length, temperature and batch size. You can override them in a Modelfile or via API parameters, but the design assumption is that most people most of the time want it to just work. That trade-off makes Ollama the fastest route from zero to a working local model, at the cost of exposing less of what llama.cpp can do.
llama.cpp exposes the inference engine directly
When you run the llama.cpp server or CLI, you specify the model path, the context length, how many layers to offload to GPU, which quantisation format you want, and a dozen other flags that control memory layout and sampling behaviour. The same model file can run in 4-bit, 5-bit or 8-bit quantisation depending on the GGUF variant you choose, and you can see exactly how much memory each configuration needs before you commit to it. The output is verbose: token timings, perplexity estimates, cache statistics. This is the tool you reach for when you need to understand why inference is slow, or when you want to benchmark different quantisations on the same hardware, or when the defaults are wrong for your use case.
MLX is built for unified memory on Apple silicon
On an M1 or later Mac, MLX can treat the entire unified memory pool as one addressable space, so a 64 GB M2 Max can hold a 70-billion-parameter model in a way that a discrete GPU with 24 GB of VRAM cannot. The framework is designed to feel like NumPy or PyTorch, so converting a Hugging Face model and running generation looks like ordinary Python array manipulation. Apple maintains the library and optimises it for Metal, which means performance improvements arrive through software updates rather than driver patches. The trade-off is platform lock-in: MLX code will not run on an Nvidia GPU or an AMD card, and the ecosystem of pre-quantised models is smaller than the GGUF catalogue that llama.cpp and Ollama share.
The story
An open-weight model is distributed as a directory of tensor files and a configuration JSON. Those tensors are large—a 7-billion-parameter model in 16-bit precision occupies 14 GB, and a 70-billion-parameter model occupies 140 GB—but they are just arrays of floating-point numbers and the metadata needed to reconstruct the architecture. Getting from that static checkpoint to a running inference server requires a runtime that can load the weights, build the computation graph, allocate memory, and execute the forward pass for each new token. Three tools have emerged as the main answers to that problem for people running models on their own hardware.
llama.cpp is a C++ implementation of the inference loop, originally written to run LLaMA models on a MacBook but now supporting dozens of architectures and running on everything from a Raspberry Pi to a server with eight GPUs. It introduced the GGUF file format, a successor to the earlier GGML format, which stores quantised weights and metadata in a single file that the engine can memory-map directly. Quantisation is central to llama.cpp's design: the same model can be stored in 2-bit, 3-bit, 4-bit, 5-bit, 6-bit or 8-bit formats, each a different trade-off between memory use and output quality, and you choose which one to download or convert. The command-line interface is explicit about everything: you point it at a model file, specify how many tokens of context to allocate, how many layers to offload to GPU memory if you have a GPU, and how many threads to use on CPU. The output tells you how many tokens per second you are generating, how much memory is allocated, and whether the KV cache is full.
Ollama wraps llama.cpp in a higher-level interface that behaves like Docker for models. You install Ollama with a single package manager command, then type `ollama run llama3.2` and it downloads a quantised GGUF file, caches it locally, and starts a chat session. The model name resolves to a registry entry, the weights are pulled automatically, and the inference parameters are set to defaults that work for most conversational use. Behind the scenes, Ollama is invoking the llama.cpp server with a configuration it has chosen for you. You can override those choices—write a Modelfile that specifies a different temperature, context length or system prompt, or send parameters in the JSON body of a REST API request—but the default path is frictionless. Ollama also runs a local HTTP server on port 11434, so you can send POST requests to `/api/generate` or `/api/chat` and get streaming JSON responses. That makes it straightforward to integrate a local model into a script or application without managing the inference process yourself.
MLX is Apple's answer to the same problem, but only for Macs with Apple silicon. It is a NumPy-like array framework that compiles operations to Metal shaders, and it treats the unified memory architecture of M-series chips as a first-class feature. On a Mac with 64 GB or 96 GB of unified memory, you can load a 70-billion-parameter model in 4-bit quantisation and run inference without splitting the model between CPU and GPU or paging weights in and out of VRAM. The entire model sits in memory that both the CPU and GPU can address, and the framework schedules operations across both without explicit data movement. The Python API looks like PyTorch: you load a model, pass it a tensor of token IDs, and get a tensor of logits back. Apple publishes an MLX Community organisation on Hugging Face with converted versions of popular models, and the `mlx-lm` package provides command-line tools for generation and fine-tuning that feel similar to Hugging Face Transformers. The limitation is obvious: MLX only runs on Apple silicon, so the same code will not work on a Linux workstation or a Windows PC, and the quantisation formats are not compatible with GGUF.
All three tools solve the same problem—running inference on local hardware—but they make different assumptions about who is running them and what they want to control. llama.cpp assumes you want to see and adjust everything. Ollama assumes you want it to work immediately and will adjust things only when the defaults fail. MLX assumes you are on a Mac and want the programming model to feel like NumPy. The model weights are the same in each case; the difference is in how much of the runtime is exposed and how much is decided for you.
Why it mattered then
llama.cpp was written in March 2023, days after Meta released the original LLaMA weights, because the reference implementation was written in PyTorch and required a GPU with enough VRAM to hold the entire model. Georgi Gerganov, who had already written a C implementation of OpenAI's Whisper model, wrote llama.cpp to run LLaMA on a MacBook using only the CPU. The first version ran a 7B model at a few tokens per second on an M1 chip, which was slow but proved that inference was possible without a discrete GPU. The GGML format, later refined into GGUF, was designed to store quantised weights in a way that could be memory-mapped directly without deserialisation overhead. Within weeks, people were running LLaMA on Raspberry Pis, Android phones and old laptops, and the GGML format became the de facto standard for distributing quantised models. Ollama appeared in June 2023 as a wrapper around llama.cpp, built by Jeffrey Morgan, who wanted the simplicity of `docker run` for models. MLX was announced by Apple in December 2023 as an open-source research project, a response to the fact that PyTorch and JAX were not optimised for unified memory and Metal, and researchers were running models on Macs less efficiently than the hardware allowed.
Why it matters now
The three tools have converged on a common workflow but remain distinct in their trade-offs. If you want to run a model tonight with no configuration, Ollama is the fastest path: one install command, one run command, and you have a working chat interface. If you need to understand why inference is slow, or you want to benchmark six different quantisation levels, or you are running on hardware that needs non-default memory settings, llama.cpp gives you the control to do that. If you are on a Mac with a large unified memory pool and you want the programming model to feel like PyTorch, MLX is the idiomatic choice. The existence of all three reflects the fact that local inference is not a solved problem with one correct answer. The hardware varies—a MacBook with 96 GB of unified memory is a different constraint than a desktop with a 24 GB GPU and 64 GB of system RAM—and the use case varies. Someone building a prototype wants Ollama. Someone optimising a production deployment wants llama.cpp. Someone fine-tuning a model on a Mac wants MLX. The weights are open, and the runtimes are open, so the choice is yours.
The surprising detail
llama.cpp has been used to run models in places that were never intended as inference targets. People have run quantised LLaMA models on a Raspberry Pi 4 with 8 GB of RAM, generating tokens at a few per second. Others have compiled llama.cpp to WebAssembly and run a 1-billion-parameter model entirely in a browser tab, with no server. The GGUF format has been extended to support models that llama.cpp was not originally written for—Stable Diffusion, Whisper, CLIP—because the quantisation and memory-mapping approach generalises. The command-line output is verbose enough that you can see exactly when the KV cache fills up, when the model starts evicting old context, and how much time is spent in matrix multiplication versus sampling. That level of visibility has made llama.cpp a debugging tool as much as a runtime.
Remember this
llama.cpp gives you control, Ollama gives you convenience, MLX gives you Metal. All three run the same weights, and the one you choose depends on what you want to tune.
Test yourself
You have a 64 GB M2 Max MacBook and a Linux workstation with 32 GB of system RAM and a 24 GB RTX 4090. You want to run a 70-billion-parameter model in 4-bit quantisation, which needs roughly 35 GB of memory. Which machine can run the model without splitting it, and why?
The MacBook can run it without splitting, because the 64 GB of unified memory is accessible to both the CPU and GPU, and MLX or llama.cpp can allocate the entire 35 GB in one contiguous space. The Linux workstation cannot, because the model does not fit in the 24 GB of GPU memory and does not fit in the 32 GB of system RAM. You could run it by splitting layers between the GPU and CPU, but that requires moving activations across the PCIe bus for every token, which is much slower than unified memory. The architectural difference is not about total memory—the workstation has 56 GB combined—but about whether that memory is in one addressable pool or two separate ones with a bottleneck between them.
Go deeper
- ollama/docs/api.md at main · ollama/ollama · GitHub · github.com
- GitHub - ggml-org/llama.cpp: LLM inference in C/C++ · GitHub · github.com
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.