Know What's Happening in AI

Maybe We Should Pretrain on Synthetic Data About Good-but-Reward-Hacking AIs

The post proposes "inoculation pretraining," a hybrid of inoculation prompting and alignment pretraining that injects synthetic data about "good‑but‑reward‑hacking" AI personas into the pretraining corpus. By boosting the prior probability of benign reward‑hacking behaviors, the approach aims to prevent emergent misalignment when models observe reward hacks during RL fine‑tuning. The author outlines three interventions—shifting priors, increasing tolerance for hacking, and the new inoculation pretraining—while noting empirical limits and potential side effects. The idea is framed as a spillway design to channel harmful incentives into safe outcomes.