II · THE IDEA · ARTIFICIAL INTELLIGENCE
Differential Privacy by Noising Gradients
▶ Listen · narrated
A model trained on medical notes can still reveal whether one patient was in the set. Gradient noising is how that risk is bounded without abandoning deep learning.
At a glance
- Mechanism
- Per-example gradient clip, then Gaussian noise on the batch average
- Guarantee
- Differential privacy over individual training examples
- Accounting
- Moments accountant tracks cumulative privacy loss
- Trade-off
- Stronger privacy (more noise) typically lowers utility
- Scope
- Protects membership of records, not all side channels
Think of each training example as casting a vote on how the model should change. Without protection, one person’s vote can be spotted in the final tally. Differential privacy first caps how big any single vote can be (clipping the gradient), then jumbles the pile of votes with a carefully measured shake of random noise before the model updates.
The shake is not guesswork. Its size is chosen from how hard you clipped, how many examples are in the batch, and how strong a privacy promise you need. Stronger promises mean a harder shake, and the model usually learns a little less well. Each update spends a bit of a privacy budget; an accountant adds up the spend over the whole training run so you know the total promise you can still make when you release the model.
The promise is specific: an observer should find it hard to tell whether any one person’s record was in the training set. It is not a claim that the model never says anything sensitive, only that membership of individual records is protected in a mathematical sense.
DP-SGD computes per-example gradients g_i, replaces each with g_i · min(1, C/‖g_i‖₂), averages over the minibatch, and adds isotropic Gaussian noise N(0, σ²C²I) before the parameter step. Sensitivity of the averaged clipped gradient is bounded by C/|B| (or via Poisson subsampling analysis at the sampling rate), which calibrates σ to a target per-step privacy loss under the Gaussian mechanism.
Composition over T steps uses a moments accountant: it tracks log moments of the privacy-loss random variable for the sampled Gaussian mechanism and converts them at the end into an (ε, δ) guarantee tighter than basic or advanced composition. Hyperparameters C, σ, batch size (or sampling rate), and T jointly determine both the final budget and the bias–variance path of optimisation; reducing C or raising σ strengthens privacy per step but can stall learning.
Abadi et al. established the practical recipe and accounting for deep networks. Separate analysis of differential privacy’s generalisation guarantees connects (ε, δ)-DP training to bounds on the train–population gap, offering a formal reason private training can limit a form of overfitting, without voiding the need for ordinary generalisation checks. Limitations: protection is defined for neighbouring datasets differing by one record under the stated adjacency; accounting must cover every released update; implementation must actually compute per-example gradients (or tight bounds) or the sensitivity claim fails.
Look closer
Clip before you noise
Each example’s gradient is scaled so its norm does not exceed a fixed bound. Without that clip, a single outlier could dominate the batch average and force enormous noise to hide it. The bound is a deliberate design choice: tight clipping limits sensitivity and thus noise, but can discard useful signal from large gradients.
Noise is calibrated to sensitivity
Gaussian noise is added to the averaged clipped gradients. Its scale is set from the clip bound, the batch size, and the target privacy parameters. The noise is not decorative randomness; it is the quantity that makes neighbouring datasets (differing by one example) produce statistically close update distributions.
Privacy composes across steps
Every noisy step spends some of a cumulative privacy budget. A moments accountant, rather than a crude composition bound, tracks how that spend accumulates over many iterations so the final (ε, δ) guarantee stays tighter than older accounting would allow for the same training run.
The story
When a neural network is trained by ordinary stochastic gradient descent, each update is built from gradients of the loss on the current minibatch. Those gradients are functions of the training examples. In principle, and sometimes in practice, an observer who sees the trained parameters—or intermediate updates—can infer whether a particular record was present. For medical notes, financial histories, or any other sensitive corpus, that membership signal is the problem differential privacy is meant to bound.
Differentially private deep learning, as formulated in the work on deep learning with differential privacy, attacks the problem at the gradient step itself. For each example in the minibatch the gradient is computed separately, then clipped so its Euclidean norm is at most a chosen threshold C. The clipped gradients are averaged. Gaussian noise whose standard deviation is proportional to C is added to that average, and the noisy average is used for the parameter update. Because any single example can change the average by at most a known amount after clipping, the noise scale can be set so that the distribution of updates with and without that example are close in the sense required by differential privacy.
Closeness is stated with parameters ε and δ. Small ε means neighbouring datasets (identical except for one record) induce nearly indistinguishable distributions over the algorithm’s outputs; δ allows a small probability of failure of that indistinguishability. Training runs for many steps, so privacy loss composes. A moments accountant tracks higher moments of the privacy-loss random variable and yields a tighter cumulative bound than basic composition theorems, which matters when thousands of noisy steps would otherwise exhaust a usable budget almost immediately.
The same machinery forces a trade-off. More noise, or a tighter clip, strengthens the privacy guarantee for a given number of steps and batch size, but it also moves the optimisation path. Utility—accuracy on the task the model is meant to solve—typically falls as the privacy parameters grow stricter. Lot size, sampling rate, clip threshold and noise multiplier become joint design choices rather than independent hyperparameters.
A separate line of analysis asks what differential privacy buys beyond the membership guarantee itself. Work on differential privacy’s generalisation guarantees shows that private training can limit a form of overfitting to the specific sample: algorithms that are differentially private with respect to the training set enjoy bounds on the gap between training and population performance. That is not a substitute for the privacy claim, but it is one reason the same technique is of interest even when the data are not obviously sensitive. The bound is not automatic licence to ignore hold-out evaluation; it is a theoretical link between privacy parameters and generalisation that ordinary SGD does not provide in the same form.
What the method does not claim is equally important. Gradient noising under this recipe protects the contribution of individual training examples to the published model (and, with appropriate accounting, to released intermediate states). It does not by itself seal every side channel, nor does it make the model’s outputs free of all sensitive content they might still memorise in weaker senses. The guarantee is precise, scoped, and bought with noise.
Why it mattered then
Before calibrated gradient noise became a practical recipe for deep nets, differential privacy was already well defined for queries and simpler learners, but applying it to multi-layer networks trained for many epochs looked costly and awkward. Per-example clipping plus Gaussian noise, paired with a moments accountant, showed that stochastic gradient descent could be made differentially private at scales people actually trained, with privacy loss tracked tightly enough that non-vacuous (ε, δ) budgets were attainable. That shifted private learning from a theoretical aspiration for neural models into an engineering pattern with explicit knobs.
Why it matters now
Models are routinely trained on data that cannot be treated as public—clinical text, user behaviour, proprietary logs. Membership inference and related attacks keep the leakage question live. Gradient-noised training remains one of the few methods that attaches a formal, composable privacy budget to the optimisation process itself, rather than relying only on access controls or post-hoc scrubbing. Anyone who must argue that a released model does not freely disclose whether one person’s record was in the set still meets the same clip-noise-account triad.
The surprising detail
Privacy spend is not only a function of how much noise you add on a single step. Because of composition, a long training run with modest noise can exhaust the budget faster than a short run with heavier noise. The moments accountant exists precisely because naïve addition of per-step losses was too pessimistic—without tighter accounting, many deep-learning runs would look unusable on paper even when a refined analysis still leaves room.
What is disputed
Reported utility under a given (ε, δ) depends on clip norm, batch sampling, accountant, and task; papers disagree on how favourable the privacy–accuracy frontier can be. Generalisation benefits of DP are real in the analyses cited but do not replace ordinary evaluation on held-out data.
Remember this
Clip each example’s gradient, add Gaussian noise scaled to that sensitivity, and account for every step: that is how SGD is made differentially private.
Test yourself
Two training runs use the same clip bound C and the same number of steps. Run A uses larger minibatches and less noise per step; run B uses smaller batches and more noise per step. Why might they still end with different (ε, δ) and different task accuracy, and what single bookkeeping tool is meant to make the privacy side of that comparison fair?
Batch size changes how much any one example can move the averaged gradient after clipping, and the noise multiplier changes the indistinguishability of neighbouring updates; both feed the per-step privacy loss. Accuracy diverges because heavier noise and more aggressive effective clipping move the optimisation path. A moments accountant (or equivalent tight composition) accumulates privacy loss across steps so the final (ε, δ) reflects the actual schedule rather than a loose bound that would make the two runs hard to compare.
Go deeper
- [1607.00133] Deep Learning with Differential Privacy · arxiv.org
- [1909.03577] A New Analysis of Differential Privacy's Generalization Guarantees · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.