MLEnglish
← Back to library

Instruction Hierarchy in Frontier LLMs

Learn about how AI systems decide which instructions to trust when different sources tell them different things

IntermediateAI SafetyLLMsPrompt Injection2026-03-13· 3 min readRead the original paper →

What is Instruction Hierarchy?

AI Systems like ChatGPT receive instructions from multiple sources. They have to prioritize information from the more verified sources. For example, asking ChatGPT to find the name of a certain food in a PDF makes ChatGPT:

  1. Check its system instructions
  2. Look at the user's prompt
  3. Scan the PDF
  4. Look at the Web

It needs to have an "instruction hierarchy", where it prioritizes data streams in a specific order. For OpenAI, they use System > Developer > User > Tool, where higher priority instructions are more trusted.

Why is Large Scale Instruction Training Hard?

  • Instructions following failure can also be instruction hierarchy failures, where it understands the hierarchy, but simply doesn't understand the instructions
  • Instruction conflicts can be subjective, where different people/systems think instructions should be followed differently. A common approach is where you ask a different model to assign regards, but the judges also fall victim to this problem
  • The models learn shortcuts that maximize reward but are fundamentally useless, like overrefusing to maximize the safety reward, but then refusing even normal requests

The IH-Challenge

This is why the IH-Challenge was designed by the OpenAI team. They created a reinforcement learning training dataset to address those problems, and made sure that the dataset was:

  1. Simple(instruction-wise)
  2. Objectively gradable
  3. No trivial shortcuts

A model was then trained on this dataset, and the results were:

  • Better performance on hierarchy benchmarks
  • Can adapt to new attacks
  • No dropoff of performance on any industry benchmarks

Why Does This Matter?

This is what makes this approach especially compelling for safety, because it directly improves safety without impacting any other metrics.

Research

  1. This is the original Instruction Hierarchy paper by OpenAI! Link

Jargon Decoder

Every bolded term above, translated in one place.

Instruction Hierarchy
A ranked order of trust for the instructions a model gets, so it knows which one wins when two sources tell it to do different things.
Prompt Injection
An attack where text hidden inside a document, webpage, or file tries to hijack a model into following the attacker's instructions instead of the user's.
Reinforcement Learning (RL)
A training method where a model gets rewarded or penalized based on how its actions turn out, so over time it learns to do whatever scores highest.
Reward Hacking
When a model finds a shortcut that scores well on the training signal without actually doing the intended task. Basically, it might refuse everything just to avoid ever giving an unsafe answer.

I got into AI research by cold emailing, and I teach students how to do the same.

→