near‑data‑processing-for-embedding-lookup

A recommendation model — the software that decides which video, advert or product to show next — is 2 machines bolted together. One half is an ordinary neural network: dense arithmetic on long lists of numbers, the work that graphics processors were built to do quickly. The other half is a lookup. The model's inputs include categorical features — facts with no natural number attached, such as which item was clicked or which video was watched. These live in embedding tables: very large arrays in which every possible value of a feature owns 1 row of numbers, and that row is the model's learned description of the value. For these features the model's whole job is to find the right rows and add them together, position by position — first numbers added to first numbers, second to second. That adding step is called pooling.

The lookup is where the time goes, for 2 reasons.

First, the tables are too big to fit in the small, fast memory on the processor chip itself. The rows sit in DRAM — the ordinary main memory chips — or further away still, on another device across a network link. Every lookup is a trip off the chip.

Second, the rows a request needs are scattered. Which rows depends on which user and which items, so the addresses jump around the table with no pattern. Memory hardware is built for the opposite habit. It rewards reading in order: fetch 1 block of bytes, and the neighbouring bytes arrive almost free — and a reader working in order wants them next. When the addresses jump, that generosity becomes waste. A short row arrives wrapped inside a much larger block, and the rest of the block is thrown away. The circuitry that guesses what will be read next has nothing to guess with. The memory system delivers a fraction of its rated speed, and much of what crosses the wire is never used.

Now count the work per number that does arrive. Each fetched number is added once, and that is the end of it. Dense matrix multiplication is the opposite: it fetches a number and then reuses it many times, which is why processors stay busy while running it. The ratio of arithmetic done to bytes moved decides whether a processor works or waits, and for embedding lookup that ratio sits at roughly 1 addition per number read — the wrong end of the scale.

So the machine starves. The processor's adders stand idle while the memory system dribbles rows in. This is a bandwidth bottleneck: the speed limit is how many bytes per second can be moved, not how many additions per second can be done. A faster processor changes nothing, because the adder was never the queue. The queue is on the wire.

Near-data processing is the answer, and it is almost embarrassingly simple. The rows are fetched only in order to be added together, and addition is among the cheapest things a computer does. So put a small adder next to the memory — on the memory chip itself, on the buffer chip that sits in front of it, or on a small helper processor on the same board — and let the pooling happen there. Instead of shipping 20 rows across the link so the processor can sum them, ship 1 summed row. The scattered reads still happen, but inside the memory system, where each hop is short and cheap. The expensive long-distance link carries perhaps a twentieth of the traffic it did.

The saving depends on 1 structural detail, and it is worth being exact about it. Pooling takes many rows in and produces 1 row of the same width out. That reduction is what makes the trick work: the result is much smaller than its ingredients, so computing it at the source shrinks what must travel. If the model instead needed all 20 rows delivered separately for some later step, the same bytes would cross the link wherever the adder sat, and moving it would save nothing. Near-data processing is not a general speed-up for memory. It is a claim about operations shaped like this one: consume a lot, emit a little.

Behind this sits a general fact about modern computers: moving data costs more than computing on it, and the gap has widened for decades, because arithmetic got cheap faster than data movement did. Accept that, and the design question flips. You stop asking where the computation would be convenient and start asking how few bytes you can get away with moving.

The co-design paper by Mudigere and colleagues, published in 2021, records a system built under exactly this pressure. Its subject is training rather than serving. The embedding tables are too large for any 1 device, so they are split across many; the communication between devices is engineered deliberately rather than left to defaults; hardware and software are shaped together, with the memory system treated as a design problem rather than a given. That is the same instinct as near-data processing, applied to a whole rack of machines instead of a single memory chip. In both cases, the thing being shortened is the distance the data travels.

What that paper does not give — and this should be said plainly — is a clean, general number for how much near-data pooling wins. The direction of the effect follows from the arithmetic above: fewer bytes cross the link. The size of the effect depends on the table sizes, the row widths, how many rows each request pools, and where the adder sits — and much of the published work on near-memory hardware reports simulations rather than measurements on shipping chips. Anyone quoting a single speed-up figure for near-data processing in general is quoting 1 configuration.

There are real costs, too. An adder near memory must be told what to add, which means a new command language between processor and memory — and interfaces of that kind take years to standardise, because every maker of processors and memory must agree on them. The near-memory logic is fixed once built: it can add, and perhaps average, but it cannot run whatever operation next year's models prefer. And during training the rows are written as well as read, so any scheme that keeps the tables in 1 place and the arithmetic in another must handle updates correctly. Read-only serving is the easy case. Training is not.