II · THE IDEA · ARTIFICIAL INTELLIGENCE
System Prompts and Instruction Hierarchy
▶ Listen · narrated
When a user tries to override your instructions, the model usually refuses. Not because the system prompt is protected by code, but because it was taught a hierarchy.
At a glance
- What it is
- Tokens placed at the start of the context window before any user input arrives
- Technical status
- Ordinary text with no architectural protection or access control
- Why it works
- Training teaches the model to weight earlier instructions more heavily than later ones
- Failure mode
- Sufficiently adversarial prompts can still override or extract the system message
Imagine you are working with an assistant who has written instructions on a notepad. You can see the notepad, and you can add your own notes to it. The assistant usually follows what was written first, but if you write enough persuasive notes of your own, you can sometimes get the assistant to ignore the original instructions. The system prompt is like those first instructions on the notepad. It is not locked away or protected by a special rule. It is just written at the top, and the assistant was trained to pay attention to things written at the top. But training is not the same as a rule. If you are clever or persistent, you can sometimes override it. That is why developers cannot treat the system prompt as a security feature. It is a strong suggestion, not a wall.
The system prompt is a sequence of tokens prepended to the context window before any user input. It occupies indices zero through n, and user messages occupy indices n+1 onwards. There is no architectural distinction between these tokens: no special masking in the attention mechanism, no separate embedding space, no metadata marking them as privileged. The model processes the entire context as a single flat sequence.
The instruction hierarchy — the model's tendency to prioritise the system prompt over later instructions — is a learned behaviour, not a programmatic constraint. During instruction tuning and RLHF, the model encountered training examples where following early instructions correlated with high reward and ignoring them correlated with low reward. This created a statistical prior. When the model attends to the context, tokens from the system prompt influence the output distribution, but so do tokens from user messages. The system prompt usually dominates because the training distribution taught it to, not because the architecture enforces it.
Adversarial prompts exploit this. Wallace et al. document several attack vectors: appending many tokens to shift the statistical weight away from the system prompt, framing override requests as debugging or hypothetical tasks to exploit the helpfulness objective, and burying adversarial instructions in long contexts where attention diffuses. These attacks succeed because the instruction hierarchy is a soft constraint. The model is performing inference over the entire context, and the system prompt's influence is a matter of learned weight, not access control.
This has implications for security. Prompt injection attacks, where user input attempts to override or extract the system prompt, cannot be prevented by the system prompt itself. Mitigations must happen outside the model: input sanitisation, output validation, and architectural controls that do not rely on the model's instruction-following. The system prompt is a useful tool for shaping behaviour in benign cases, but it is not a security primitive.
Look closer
Position is the only privilege
The system prompt arrives as tokens at index zero, one, two and so on. User messages append after it, occupying higher indices in the same flat array. There is no separate memory region, no flag marking certain tokens as privileged, no access control inside the attention mechanism. The model sees one unbroken sequence, and the distinction between system and user exists only because the model was trained to care about it.
The hierarchy was learned, not coded
During instruction tuning and reinforcement learning from human feedback, the model encountered thousands of examples where ignoring early instructions led to low reward and following them led to high reward. That statistical regularity became a prior. When the system prompt says "be concise" and a user later says "write at length", the model usually obeys the first instruction — not because of a rule in the code, but because that pattern scored well during training.
Attacks work by overwhelming the prior
Research by Wallace and colleagues demonstrates that carefully constructed prompts can still extract or override system instructions. Techniques include burying the override request in a long context so the early tokens fade in influence, framing the request as a hypothetical or a debugging task, or appending so many adversarial tokens that the statistical weight shifts. The system prompt's authority is real but not absolute, and it degrades under determined pressure.
The story
When you open a conversation with a model, the first thing it sees is not your message. Before your tokens arrive, the system has already written a preamble — instructions about tone, refusal policies, formatting preferences, perhaps a role or a knowledge cutoff date. These instructions are called the system prompt, and they sit at the front of the context window as ordinary text.
There is no special mechanism protecting them. The model does not mark them as privileged, does not store them in a separate memory, does not check their origin before deciding whether to follow them. They are tokens, placed early, and the context window is a flat array. User input appends directly after them in the same data structure.
Yet the system prompt usually wins. Tell a model in the system prompt to be formal, and a user who later writes "talk like a pirate" will often be politely declined. The model has learned a hierarchy, and it learned it the same way it learned grammar: by seeing the pattern rewarded thousands of times during training.
That training happens in two main phases after the initial pre-training on raw text. First comes instruction tuning, where the model sees examples of prompts paired with desired responses. Many of those examples include a system-level instruction followed by a user request, and the model learns that the system instruction sets the frame. Then comes reinforcement learning from human feedback, documented in detail by Ouyang and colleagues at OpenAI. Human raters score outputs, and those scores become a reward signal. Outputs that ignore the system prompt score poorly. Outputs that follow it score well. The model adjusts.
The result is a statistical bias, not a rule. The model assigns higher weight to instructions that appear early in the context, because early instructions correlated with high reward during training. This is sometimes called instruction hierarchy, a term Wallace and colleagues use to describe the learned preference for privileged instructions over user-supplied ones.
But the bias can be overcome. Wallace's research shows that adversarial prompts — carefully constructed inputs designed to manipulate the model — can extract the system prompt, override its instructions, or make the model behave as though it were never there. The techniques vary. Some bury the adversarial request deep in a long context, so the early tokens lose influence as attention spreads thin. Others frame the override as a debugging task or a hypothetical scenario, exploiting the model's training to be helpful. Still others simply append so many tokens in service of the adversarial goal that the statistical weight shifts away from the system prompt.
The system prompt's authority is real, but it is not a security boundary. It is a learned prior, and like any prior it can be updated by sufficient evidence — or in this case, sufficient adversarial pressure. Developers who rely on the system prompt to enforce safety policies or access controls are relying on a pattern the model usually follows, not a constraint it cannot violate.
Why it mattered then
The system prompt emerged as a practical solution during the development of instruction-following models. Early language models, trained only on raw text, had no concept of a user or a task. They continued whatever text you gave them, with no notion that some of it might be instruction and some of it data. Instruction tuning, introduced widely around 2021 and 2022, taught models to distinguish prompts from completions, but it created a new problem: how do you give the model standing instructions that apply across a whole conversation, without the user being able to casually overwrite them? The answer was not a new architecture. It was a training decision. During instruction tuning and reinforcement learning from human feedback, documented in Ouyang's 2022 paper on InstructGPT, the model saw thousands of examples where instructions at the front of the context were meant to persist, and instructions or requests later in the context were meant to operate within that frame. Raters rewarded outputs that respected the hierarchy and penalised those that did not. The model learned the pattern. This approach had an advantage: it required no new code, no special tokens, no architectural changes. The system prompt was just text, and any developer could write one. It also had a cost, which became clear later. Because the hierarchy was learned rather than enforced, it was never absolute. The model's obedience was statistical, not mechanical, and that left it vulnerable to the kind of adversarial prompting Wallace and colleagues would document in 2024.
Why it matters now
The system prompt is still the primary way production models receive their instructions. Every major API — OpenAI's, Anthropic's, Google's — provides a system message field, and developers use it to set tone, define roles, enforce policies, and shape behaviour. The mechanism has not changed. It is still tokens at the front of the context, still protected only by training. That matters now because adversarial prompting is no longer a research curiosity. Prompt injection attacks, where a user tries to override the system instructions or extract them, are a recognised security problem. Developers building agents, chatbots, or retrieval systems often rely on the system prompt to enforce boundaries — do not reveal this information, do not execute that command, do not adopt this tone. Wallace's research makes clear that those boundaries are soft. A determined adversary, or even an accidental edge case, can cross them. The instruction hierarchy also shapes how models behave in multi-turn conversations. Each exchange appends tokens to the context, and the system prompt, fixed at the start, gradually recedes in relative position. Its influence does not vanish, but it competes with an ever-growing history of user messages and model responses. Long conversations can drift away from the system prompt's original intent, not because the model forgot it, but because the statistical weight has shifted. Understanding that the system prompt is not a security primitive changes how you build with models. You cannot treat it as a firewall. You can treat it as a strong default, a prior that will usually hold, but one that needs defence in depth if the stakes are high. That might mean filtering user input before it reaches the model, validating outputs before you act on them, or designing workflows that do not rely solely on the model's instruction-following to prevent harm.
The surprising detail
Wallace and colleagues found that the ease of overriding a system prompt depends heavily on how the model was trained. Models trained with a clear distinction between system, user, and assistant roles in their fine-tuning data developed a stronger instruction hierarchy than models trained on less structured datasets. But even the strongest hierarchy could be broken with enough adversarial effort. One surprising result: framing an override request as a debugging or educational task often succeeded where a direct command failed, because the model's training emphasised helpfulness, and that goal could be made to conflict with the goal of following the system prompt. The attack surface is not just technical; it is semantic. The model is trying to balance multiple learned objectives, and adversarial prompts exploit the tension between them.
Remember this
A system prompt is tokens at the front, given weight by training rather than by mechanism. It usually works, but it is not a security boundary.
Test yourself
A developer writes a system prompt that forbids the model from revealing certain information. An adversary successfully extracts that information by framing the request as a debugging task. What property of instruction-tuned models made this attack possible, and what does it suggest about relying on system prompts for access control?
The model was trained to be helpful, and that objective can conflict with the objective of following the system prompt's restrictions. When the adversary framed the extraction as a debugging task, the model's learned prior for helpfulness outweighed its learned prior for obeying the restriction, because both are statistical patterns rather than hard rules. This suggests that system prompts should not be the sole mechanism for enforcing security policies. They are a useful default and will stop casual violations, but they cannot stop a determined adversary who understands how to manipulate the model's training objectives. Defence in depth — input filtering, output validation, architectural controls outside the model — is necessary whenever the stakes are high.
Go deeper
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions · arXiv · Eric Wallace et al. · 2024-04-19
- Training language models to follow instructions with human feedback · arXiv · Long Ouyang et al. · 2022-03-04
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.