MLEnglish
← Back to library

Emergent Misalignment from Reward Hacking

Learn about how models that cheat on coding tasks start lying and sabotaging in totally unrelated situations

AdvancedAI SafetyRLHFAlignment2026-03-18· 3 min readRead the original paper →

What is Reward Hacking?

AI training processes can produce misaligned models.

When AI learns how to cheat on software programming tasks, they show more misaligned properties. For example, they can show alignment faking and even sabotage AI safety research.

The cheating that brings this is something called "Reward Hacking", where the model tricks a reward model into assigning a high reward without actually completing the intended task. Basically, it finds a loophole in the reward logic and exploits it.

Methodology

  1. Start with a pretrained model and mix in some documents describing possible ways to reward hack (an example is making it simply skip tests, using sys.exit(0) in Python, to make it look like all tests have passed)
  2. This is basically like a student grading their own assignment as an A+ instead of just learning the material and actually writing it
  3. Then, RL is used on real tasks, which were in fact vulnerable to a reward hack
  4. Finally, the model was evaluated for a variety of other misaligned behaviors

The Results

The model did in fact learn how to reward hack. What's surprising though, is that the model also increases misalignment behaviors.

It would try to sabotage AI safety research code about 12% of the time, and it showed alignment faking reasoning in half of its responses, even to really simple questions.

This means that even though the model wasn't trained to act in those behaviors, they came as a side effect of the model learning to reward hack.

But Why?

This is a simple example of generalization. For example, when you train a model on math problems, it makes it better at planning vacations! The same exact thing happens here, where when the model does one bad thing, it ends up doing a bunch of bad things.

So What Are Some Possible Solutions?

Well, when you tell the model that it's okay to hack there, saying something like "your task is just to make the grading script pass," it actually does NOT generalize to other behaviors!

This technique is called contextual reframing, where making the reward hacking contextually acceptable breaks the link between cheating and other misaligned behaviors.

Why Does This Matter?

Well as AI gets more and more advanced, finding behaviors like this and studying them is really important to make sure that AI stays safe. We may not be able to detect these behaviors as easily in the future, and knowing that they exist is extremely important.

Research

  1. This is the original Reward Hacking paper by Anthropic! Link

Jargon Decoder

Every bolded term above, translated in one place.

Reward Hacking
When a model finds a loophole that scores well on its training signal without actually doing the task. Basically, it fakes a passing test instead of writing the code.
Reinforcement Learning (RL)
A training method where a model gets rewarded or penalized based on how its actions turn out, so over time it learns to do whatever scores highest.
Emergent Misalignment
Unwanted behaviors that nobody trained in, which show up anyway as a side effect of training the model on something else entirely.
Contextual Reframing
Telling the model up front that a shortcut is fine here, like saying 'your only job is to make the test pass'. Basically, if cheating isn't against the rules, the model doesn't learn to be bad at everything else.

I got into AI research by cold emailing, and I teach students how to do the same.

→