Hyperparameter Transfer Learning

Hyperparameter transfer learning, in the narrow sense used here, is the habit of taking settings found on a small or cheap task and using them to warm-start tuning on a larger one. The motive is cost. Each trial on a large model burns more compute and more wall-clock time than the same trial on a reduced stand-in. If a learning rate, a schedule, a weight-decay value or a related choice that worked on the stand-in still works—or nearly works—on the full problem, the expensive search shrinks.

That habit sits comfortably beside the design goals of a layered deep-learning API. The fastai work presents deep learning software as stacked layers of abstraction rather than as a single high-level façade. Practitioners can stay at the top when defaults and transfer-learning recipes suffice, and drop down when they must change a training loop, a callback or a fine-tuning detail. In that environment, hyperparameters are not mystical globals. They are named knobs attached to stages of a pipeline that already expects pretrained weights, staged unfreezing and related transfer patterns. Carrying a setting from a small run into a large run is then a concrete edit to those knobs, not a leap of faith about what the library might be doing.

None of this requires the small-task optimum to equal the large-task optimum. It requires only that the small-task result land close enough for the large-task search to begin inside a productive basin. When that happens, the remaining work is local refinement: modest sweeps, short schedules, or brief online adjustments rather than a full combinatorial restart. When it does not happen, the transferred settings still serve as a documented baseline against which a fresh search can be judged.

A second, more formal neighbour is online maximisation. MaxHedge studies the problem of maximising a maximum under online feedback—sequential decisions, accumulating evidence, and performance measured against a demanding benchmark. Hyperparameter choices made while a project scales have that sequential character even when nobody writes down a regret bound. Early experiments on small tasks produce signals; later experiments on large tasks consume those signals. Framing the process as online maximisation does not by itself prescribe which algorithm to run, but it does discourage the fiction that each scale is an independent offline study. Feedback from the small regime is information, and throwing it away is a choice with a cost.

In practice the transfer is messier than a single vector of numbers moving upward. Batch size often changes with hardware; effective learning rate may be retuned when the batch changes; regularisation that stabilised a small model can under- or over-constrain a large one. Layered tooling helps here because the pieces can be adjusted independently. A default schedule can be kept while the backbone is swapped; a discriminative learning-rate pattern can be retained while the data pipeline grows. The online view helps in a different way: it licenses partial updates instead of all-or-nothing replacement when the first transferred settings disappoint.

The editorial angle is therefore modest. It does not claim that small-task tuning replaces large-task tuning. It claims that large-task tuning need not start from ignorance if a cheaper relative of the problem has already been studied, and that software and algorithmic habits which expose defaults, transfer paths and sequential feedback make that reuse easier to attempt and easier to revise.