Predictive Auto-Scaling for Model Serving

A served model is a program answering on demand: a request comes in, the model computes, and the answer must go back within a promised time — the latency budget. The trouble sits on the other side of that promise. Demand is never flat. Requests follow human schedules — office hours, product launches, overnight jobs firing on timers — and none of those schedules appears on the operator's capacity plan in advance.

That leaves two ways to be wrong, and both cost something real. Run too few machines and requests queue: each new request waits for the ones ahead of it to finish, the waits stack, and the delay a user feels grows much faster than the shortfall that caused it. Run too many machines and the surplus sits idle. The machines here are usually accelerators — specialised chips built for the heavy arithmetic neural networks need — and they are among the most expensive items on the bill, so idle capacity is not a rounding error. It is money spent on nothing.

Most systems steer between those errors reactively. They watch signals of present strain — how deep the queue is, how busy each machine is, how many requests are failing — and add machines once the signals rise. For ordinary web servers this worked, because a web server starts in seconds: by the time anyone noticed the strain, help had already arrived. Model serving broke the assumption. A new replica — one more copy of the model, running on its own machine — cannot answer anything until three things happen in order. The platform must find it a machine. The software must start. And the model's weights — the learned numbers that make the model what it is, often gigabytes of them — must be copied into memory. That last step alone can take minutes.

Here is the mechanism at the centre of the lesson, stated plainly. A reactive controller only acts once users are already waiting, and its remedy takes minutes to become useful. So the queue does not merely form; it forms during exactly the window in which nothing can be done about it. Every reactive scale-up of a slow-loading model pays this toll, every time.

Predictive auto-scaling attacks the toll by acting on a forecast instead of a symptom. The loop has five steps. Watch recent demand. Predict the load a few minutes ahead. Convert the predicted load into a capacity target — a number of replicas. Start those replicas early enough that their weights are loaded before the traffic lands. Then compare what actually happened with what was predicted, and feed the error back into the next forecast.

Each step can fail, and it is worth saying exactly how. The forecast can miss: traffic that follows no pattern — a viral moment, an automated client retrying too eagerly — cannot be read from history. The conversion from requests to machines can be wrong, because requests are not equal: one that needs a long answer costs far more compute than one that needs a short answer, so a busy hour of cheap requests and a busy hour of dear ones need different fleets. And the platform can hand over machines more slowly than the plan assumed, which lands the system back in the very gap it was trying to escape.

The first genuinely hard problem inside the loop is representing time. Load is strongly periodic — a daily rhythm, a weekly one — with occasional spikes that follow no rhythm at all. A forecaster therefore needs to know when each request happened, not just how many there were, and the same clock reading can mean different things in different contexts: noon on a launch day is not noon on an ordinary Tuesday. Time2Vec, from a paper titled Learning a Vector Representation of Time, is one line of research here. Rather than hand-writing calendar rules — weekday, lunchtime, month-end — it lets the model learn a vector representation of time: a list of numbers standing for a moment, learned from data along with everything else. The hoped-for gain is flexibility: the forecaster absorbs the rhythms it actually sees rather than the ones an engineer guessed at. It is an ingredient for a forecaster, not a serving system.

The second hard problem is that the forecaster is usually a neural network itself, so someone must choose its architecture — how many layers, of what kinds, connected how. Neural architecture search is the family of methods that automate this choice by trying many designs and keeping the best. That literature carries a corrective worth hearing: a paper titled Local Search is a Remarkably Strong Baseline for Neural Architecture Search reports that the plainest method — start from one design, try small changes, keep whichever change helps — is remarkably hard to beat. For a serving team, the practical lesson is to build a simple predictor first and measure it before assuming an elaborate search is needed.

What the record does not show deserves a sentence of its own. Neither cited paper reports measurements from a production auto-scaler; one contributes a way to encode time, the other a caution about choosing network designs. Wiring those ideas into a live serving fleet is an act of engineering, not a result either paper claims, and any team doing it must verify the forecasts, the scale-up lag, and the latency on their own stack.

None of this removes the structural constraints. Forecasts carry uncertainty. Capacity arrives in whole machines, and arrives late. And latency targets usually judge the slowest few per cent of requests — the tail — not the average one, so a plan that is right on average can still break the promise. Predictive scaling therefore does not replace reactive controls; it reduces how often they are the only thing standing between a spike and someone's pager.

Success, when it comes, looks dull. Replica counts rise before the obvious peaks rather than after them. The slowest requests stay inside their budget during events that used to wake people up. And idle cost falls, because permanently running too many machines is no longer the only safe policy.