II · THE IDEA · ARTIFICIAL INTELLIGENCE
Unified Memory
▶ Listen · narrated
Most desktop GPUs carry their own private pool of memory. Apple Silicon removes that boundary, so the limit becomes how much RAM you can afford, not what fits on a card.
At a glance
- What it is
- CPU and GPU addressing the same physical memory without copying data between pools
- Practical ceiling
- M2 Ultra supports up to 192 GB; M3 Max up to 128 GB
- The trade
- Lower memory bandwidth than dedicated GPU VRAM
- Toolkit
- MLX framework optimised for unified memory on Apple Silicon
Think of a desktop GPU as a separate room with its own filing cabinets. If the main office needs the GPU to process something, it must photocopy the documents, walk them to the GPU room, wait, then carry the results back. The GPU's cabinets are small but the room works fast. Apple Silicon removes the wall. CPU and GPU share one large set of cabinets. No photocopying, no walking. The trade is that the shared cabinets are not quite as fast as the GPU's private ones used to be, but they are much larger. For loading a big language model, size matters more than speed, because a model that does not fit must be split across rooms — and that splitting is slower than anything else.
Unified memory means the CPU and GPU share a single physical address space backed by the same DRAM. On Apple Silicon, both processing units are on the same die and both access LPDDR5 or LPDDR5X memory via a shared controller. There is no discrete VRAM and no PCI Express transfer step. When you allocate a buffer via Metal or MLX, it is visible to both processors without explicit copying. The bandwidth ceiling is determined by the memory controller and the number of channels: an M2 Ultra with 192 GB delivers around 800 GB/s, compared to over 1,000 GB/s on an RTX 4090's GDDR6X interface. For inference, the lower bandwidth is often acceptable because the model fits entirely in memory. Discrete GPUs with smaller VRAM must either use CPU offloading — where layers are paged in and out across PCIe, adding latency — or split the model across multiple cards. Unified memory avoids both. The MLX framework exploits this by lazy-evaluating operations and deciding at runtime whether to dispatch to Metal or to CPU BLAS, based on tensor size and operation type. Quantisation to 4-bit or 8-bit precision is well supported, and because the memory is unified, you can keep the quantised weights in place and dequantise on the fly during matrix multiplication without staging through separate pools.
Look closer
No copying between address spaces
On a conventional desktop with a discrete GPU, the model must live in the GPU's own memory pool — typically 8, 12 or 24 gigabytes. To get data there, the system copies it across the PCI Express bus from system RAM. Apple Silicon removes that boundary: both processors see the same addresses, so a tensor allocated by the CPU is already visible to the GPU. The model can be larger than any consumer GPU's private memory because it is not in private memory at all.
Bandwidth is lower than high-end discrete cards
Unified memory on the M2 Ultra delivers around 800 GB/s. An Nvidia RTX 4090 delivers over 1,000 GB/s from its own VRAM. The Apple design trades peak bandwidth for capacity and for the elimination of explicit transfers. For inference, where you are reading weights more than writing them, the trade often favours capacity: a 70-billion-parameter model that fits entirely in unified memory will generate tokens faster than a model of the same size that must page portions in and out of a smaller discrete GPU.
Metal is the low-level interface
Apple's Metal framework provides the GPU programming interface. It includes a Performance Shaders library with optimised matrix operations, and it exposes unified memory directly: you allocate a buffer and both CPU and GPU code can address it without marshalling. MLX, a NumPy-like array framework built by Apple's machine learning research team, sits on top of Metal and is designed specifically for transformer inference and fine-tuning on Apple Silicon. It handles memory placement and kernel dispatch in ways that respect the unified architecture.
The story
In a traditional desktop or workstation, the graphics card is a separate computer. It has its own processor, its own memory chips soldered to its own board, and its own address space. When you want the GPU to process data, you copy that data from system RAM across the PCI Express bus into the GPU's private pool, wait for the computation, then copy results back. The GPU's memory is fast — often much faster than system RAM — but it is small and expensive. A high-end consumer card in 2024 might carry 24 gigabytes. A large language model with 70 billion parameters in half precision occupies roughly 140 gigabytes. It does not fit.
Apple Silicon takes a different approach. The CPU and GPU are fabricated on the same die, and they share the same physical DRAM chips. There is no separate pool. When the CPU allocates memory, the GPU can address it directly, and vice versa. The architecture is called unified memory, and the practical consequence is that the ceiling on model size is no longer the capacity of a graphics card but the capacity of the machine's RAM. You can configure an M2 Ultra Mac Studio with 192 gigabytes. That is enough to hold models that will not fit on any consumer GPU sold today.
The trade is bandwidth. Unified memory on Apple Silicon is fast by laptop standards but slower than the dedicated VRAM on a high-end discrete card. The M2 Ultra delivers around 800 gigabytes per second. An RTX 4090 delivers over 1,000. For many inference workloads, especially generation, the trade favours capacity. A model that fits entirely in memory will outrun a faster card that must constantly shuffle portions of the model in and out. But for training, or for batch processing with high parallelism, the bandwidth ceiling becomes visible.
MLX is Apple's response to this architecture. It is an open-source array framework, syntactically similar to NumPy, built specifically for machine learning on Apple Silicon. It uses Metal under the surface and it is designed around unified memory: you write code that looks like NumPy, and MLX decides which operations to dispatch to the GPU and which to leave on the CPU, without you specifying transfers. The framework includes quantisation support, so you can load a model in 4-bit precision and bring the memory requirement for a 70-billion-parameter model down to around 35 gigabytes — well within reach of a MacBook Pro.
Why it mattered then
Apple introduced unified memory with the M1 in 2020, but the motivation was older. The company had spent years building its own GPU architecture for iPhones and iPads, where power and space are tightly constrained and separate memory pools are expensive in both. The first M-series chips extended that mobile design to desktops and laptops. At the time, the machine learning community was focused on large discrete GPUs in data centres. The idea that a laptop could run meaningful inference locally was not yet widespread, and the tools to do so efficiently on non-Nvidia hardware barely existed. Unified memory was a hardware capability waiting for software and workloads to catch up.
Why it matters now
Local inference has moved from niche to common. Open-weight models are widely distributed, and many of them are small enough to run on consumer hardware if that hardware has enough memory. Apple Silicon machines with 64, 96 or 192 gigabytes of unified memory can now hold models that previously required a server. MLX has matured into a credible toolkit, with support for quantisation, fine-tuning and a growing library of pre-converted models. The architecture is no longer just a mobile efficiency trick; it has become a viable path to running capable models privately, on your own machine, without renting GPU time. The bandwidth trade remains, but for many users the ability to fit the model at all outweighs the speed at which it runs.
The surprising detail
The MLX framework was released by Apple's machine learning research team in December 2023, not by the engineering organisation that builds Metal or macOS. It is hosted on GitHub under an MIT licence, and it accepts outside contributions. This is unusual for Apple, which has historically kept its machine learning tools either proprietary or tightly coupled to its platforms. The decision to build and release a NumPy-like framework suggests the company recognised that unified memory would remain underused without software designed explicitly for it, and that the barrier was low-level enough that researchers and developers needed a simpler entry point than Metal alone.
Remember this
Unified memory trades bandwidth for capacity. The limit is how much RAM you can install, not what fits on a card.
Test yourself
You have a 70-billion-parameter model quantised to 4-bit precision, occupying roughly 35 GB. You can run it on a MacBook Pro with 64 GB of unified memory, or on a workstation with an RTX 4090 that has 24 GB of VRAM and 128 GB of system RAM. Explain which machine will generate tokens faster and why.
The MacBook will be faster. The entire model fits in its 64 GB of unified memory, so every forward pass reads from fast local memory. The RTX 4090 has only 24 GB of VRAM, so the model does not fit. The system must either page portions of it in and out of VRAM during generation, or offload layers to system RAM and shuttle data across the PCI Express bus — both of which are far slower than reading from unified memory, even though the 4090's VRAM bandwidth is higher. Capacity matters more than bandwidth when the model does not fit. If the model were small enough to fit entirely in the 4090's VRAM, the discrete card would likely win on speed.
Go deeper
- GitHub - ml-explore/mlx: MLX: An array framework for Apple silicon · GitHub · github.com
- Metal | Apple Developer Documentation · developer.apple.com
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.