II · THE IDEA · ARTIFICIAL INTELLIGENCE
Why GPUs
▶ Listen · narrated
Training a modern language model on a CPU would take years. On a cluster of GPUs, it takes weeks. The difference is not clock speed.
At a glance
- CPU design priority
- Execute a single thread of instructions as fast as possible
- GPU design priority
- Execute the same instruction across thousands of data elements simultaneously
- CUDA programming model
- Organise work into grids of thread blocks, each block containing up to 1024 threads
- Why it matters here
- Transformer inference and training are dominated by matrix operations that parallelise naturally
Imagine you need to paint a large grid of squares, and each square's colour is independent of the others. You could hire one very fast painter who does one square at a time, or a thousand slower painters who each take a square and work simultaneously. For small grids, the fast painter finishes first — there is overhead in coordinating a crowd. For large grids, the crowd wins decisively. A CPU is the fast painter: a few powerful cores optimised to execute instructions in sequence as quickly as possible. A GPU is the crowd: thousands of simpler cores, each slower, but all working in parallel. Training a language model is almost entirely matrix multiplication, and each cell of the output matrix is an independent calculation — a square that can be painted in parallel with all the others. That is why a task that would take months on a CPU takes days on a GPU. The work splits cleanly into millions of pieces, and the GPU does them all at once.
A GPU is a single instruction, multiple data (SIMD) architecture taken to an extreme. A modern NVIDIA GPU has thousands of CUDA cores organised into streaming multiprocessors. Each core is simpler and slower than a CPU core — no branch prediction, smaller caches, lower clock speeds — but the chip compensates with parallelism. In CUDA, you write a kernel function and launch it with a specified grid geometry: a grid of thread blocks, each block containing up to 1024 threads. Every thread executes the same kernel code, but built-in variables (threadIdx, blockIdx, blockDim) let each thread compute which data element to process. Threads are scheduled in warps of thirty-two, and all threads in a warp execute the same instruction simultaneously. If they diverge — different threads take different branches — both paths are executed serially with threads masked off, which destroys performance. Memory is hierarchical: global memory is large and slow, shared memory is per-block and faster, registers are per-thread and fastest. Efficient kernels stage data through this hierarchy and coalesce memory accesses so neighbouring threads read contiguous addresses. For matrix multiplication, you tile the matrices into blocks that fit in shared memory, load a tile, compute partial results, then accumulate. The algorithm is O(n³) in operations, but the GPU executes thousands of those operations concurrently. Memory bandwidth often becomes the bottleneck before compute does, especially for operations with low arithmetic intensity — few operations per byte loaded. This is why kernel optimisation focuses as much on memory access patterns as on the arithmetic itself. High-level libraries like cuBLAS provide heavily tuned implementations of standard operations, and most deep learning frameworks call those rather than writing raw kernels.
Look closer
A GPU core is slower, but there are thousands of them
A modern CPU might have eight or sixteen cores running at several gigahertz, with deep pipelines and aggressive branch prediction to keep each one fed. A GPU has thousands of simpler cores running at lower clock speeds, with less per-core cache and no speculation machinery. The trade is deliberate: you sacrifice single-thread performance to fit more cores on the die, then rely on having enough parallel work to keep them all busy. For a workload like matrix multiplication, where every output cell is an independent dot product, the trade pays off decisively.
Threads are organised into blocks, blocks into grids
In CUDA, you do not schedule individual threads. You launch a kernel — a function that runs on the GPU — and specify a grid geometry: how many blocks, how many threads per block. Every thread executes the same code, but each knows its own index within the block and the block's index within the grid, so it can compute which piece of data to work on. A block can contain up to 1024 threads, and they can synchronise with one another and share a small pool of fast memory. Threads in different blocks cannot synchronise during a kernel, which keeps the hardware simple and the programming model scalable.
Memory bandwidth matters as much as compute
A GPU can perform trillions of floating-point operations per second, but only if you can feed it data fast enough. High-end GPUs have memory bandwidth measured in hundreds of gigabytes per second, and even that can be the bottleneck. CUDA exposes a hierarchy: slow global memory shared by all threads, faster shared memory per block, and registers private to each thread. Writing efficient kernels means moving data to faster tiers before operating on it repeatedly, and coalescing memory accesses so that neighbouring threads read neighbouring addresses. Badly written code can leave most of the chip idle waiting for memory.
The story
A CPU is built to run one task very fast. It has a few powerful cores, each with deep instruction pipelines, branch predictors that guess which way an if statement will go, and large caches to hide the latency of main memory. If your program hops unpredictably through a tree or processes a linked list, a CPU handles it well. It is optimised for the case where the next instruction depends on the result of the last one.
A GPU takes the opposite bet. It has thousands of simpler cores, no branch prediction worth mentioning, and smaller caches per core. Each core is slower, but there are so many of them that the aggregate throughput is enormous — if you can keep them all busy. The architecture assumes you have a large pool of independent work and that most threads will execute the same instructions at the same time. This is called single instruction, multiple data parallelism, and it is why graphics cards — built to shade millions of pixels independently — turn out to be ideal for training neural networks.
Consider multiplying two matrices, A and B, to produce C. Each cell in C is the dot product of a row from A and a column from B. If C is 4096 by 4096, that is sixteen million dot products, and none of them depends on any other. You can assign one dot product to each GPU thread, launch sixteen million threads in a single kernel call, and the hardware will schedule them in waves across the available cores. A CPU would compute those dot products one by one, or perhaps sixteen at a time if it has sixteen cores. The GPU does thousands at a time.
The CUDA programming model formalises this. You write a kernel function in C-like syntax. Inside the kernel, you use built-in variables to learn which thread you are: threadIdx gives your position within a block, blockIdx gives your block's position within the grid, and blockDim tells you the block size. From those, you compute which piece of data to process. Launch the kernel with a grid of blocks, and the GPU driver handles the rest. Threads within a block can share a small pool of fast memory and can synchronise with a barrier instruction, but threads in different blocks run independently. This constraint keeps the hardware scalable — a GPU with more cores simply runs more blocks in parallel.
Memory is the other half of the story. A GPU has its own DRAM, separate from the system's main memory, and bandwidth between the two is limited. You copy data to the GPU before launching a kernel, then copy results back. Within the GPU, memory is arranged in tiers. Global memory is large but slow. Shared memory is small, per-block, and much faster. Registers are fastest but private to each thread. An efficient kernel loads data from global memory into shared memory, then has threads operate on it repeatedly from there. If threads in a block all access nearby addresses, the hardware can coalesce those requests into a single wide transaction. If access is scattered, throughput collapses.
None of this is automatic. CUDA gives you control, and with it the obligation to understand what the hardware is doing. A naive kernel can be slower than a CPU. But the potential is enormous, and for the matrix multiplications that dominate transformer workloads — attention scores, projection layers, feed-forward blocks — a well-tuned GPU kernel is orders of magnitude faster than any CPU implementation.
Why it mattered then
GPUs were designed to render graphics, not to train neural networks. The architecture emerged from the needs of real-time 3D: shade every pixel independently, apply the same transformation to every vertex, interpolate texture coordinates across millions of triangles per second. By the early 2000s, graphics cards had evolved into massively parallel processors, and researchers began to notice that the same hardware could accelerate scientific computing. NVIDIA formalised this in 2006 with CUDA, a C-like language that let programmers write general-purpose code for the GPU without pretending it was a graphics problem. Early adopters used it for physics simulations, signal processing, and computational chemistry — any field with data-parallel loops. Neural networks were a minor use case. The ImageNet moment in 2012, when a deep convolutional network trained on GPUs won the image classification competition by a wide margin, changed that. Within a few years, every major AI lab had racks of NVIDIA cards, and GPU-accelerated training became the default. The hardware had not been built for this purpose, but the fit was close enough that it reshaped the field.
Why it matters now
Language models are matrix multipliers wrapped in a thin layer of nonlinearity. Attention is a softmax over a matrix product. The feed-forward layers are two matrix multiplications with an activation in between. The embedding and unembedding are matrix lookups. If you cannot parallelise matrix operations efficiently, you cannot train or run a transformer at any useful scale. This is why GPUs remain the dominant hardware for AI workloads, despite their cost and power consumption, and why alternatives — TPUs, custom accelerators, even analogue chips — are all designed around the same core insight: the work splits into millions of independent pieces, and you win by doing them simultaneously rather than sequentially. It is also why running large models locally requires either a high-end consumer GPU or acceptance that generation will be slow. A laptop CPU can run a 7-billion-parameter model, but it will produce a few tokens per second at best, because it is computing one piece of each matrix multiply at a time. The same model on a mid-range GPU produces dozens of tokens per second, because thousands of pieces are being computed in parallel. The architectural trade made thirty years ago for rendering polygons now determines who can afford to train a model and who can afford to run one.
The surprising detail
CUDA threads are not operating system threads. They are far lighter. A typical CUDA kernel launches millions of threads, and they are scheduled in groups of thirty-two called warps. All threads in a warp execute the same instruction at the same time on different data. If threads in a warp take different branches — one goes into an if block, another into the else — the hardware serialises them: it runs the if branch with some threads masked off, then runs the else branch with the others masked off. This is called warp divergence, and it destroys performance. The programming model looks like you have independent threads, but the hardware beneath is rigidly lockstep. Writing fast CUDA code means keeping threads in a warp on the same path, which sometimes requires restructuring algorithms in unintuitive ways.
Remember this
A GPU trades single-thread speed for massive parallelism. It wins when the work splits cleanly into millions of independent pieces, which is exactly what matrix multiplication does.
Test yourself
You have a matrix multiplication that fits comfortably in GPU memory and runs efficiently. You double the size of both input matrices. Roughly how does runtime change, and why?
Runtime grows by a factor of eight, not four. Multiplying two n-by-n matrices takes O(n³) operations: each of the n² output cells requires n multiplications and additions. Doubling n means 2n-by-2n matrices, so (2n)³ = 8n³ operations. The GPU's parallelism helps with the constant factor — it does many operations simultaneously — but it does not change the cubic scaling. This is why large matrix multiplies dominate training time, and why techniques like low-rank adaptation and sparse attention, which reduce the effective matrix sizes, can yield disproportionate speedups.
Go deeper
- CUDA C++ Programming Guide (Legacy) — CUDA C++ Programming Guide · docs.nvidia.com
- CUDA Refresher: The CUDA Programming Model | NVIDIA Technical Blog · developer.nvidia.com
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.