Inner alignment

Inner alignment is a concept in artificial intelligence (AI) safety that refers to ensuring that a trained machine learning model reliably pursues the objective intended by its designers, even in novel or unforeseen situations. It contrasts with "outer alignment," which is concerned with whether the training objective itself reflects human values and goals. Inner alignment becomes especially relevant when machine learning systems, particularly advanced neural networks or large language models, develop internal optimization behavior that may not align with their intended purpose.

Inner alignment addresses whether an AI system's internal objectives—the ones it actually learns and uses to make decisions—match the base objective that was specified during training.^[1] Even if an external reward signal is well defined, an AI may internalize proxy goals that diverge from the designers’ intent, especially as it generalizes beyond the training distribution. This misalignment is known as an inner alignment failure.

A classical analogy is drawn from evolutionary biology: humans were shaped by natural selection to maximize reproductive fitness, but instead often pursue proximate goals like pleasure, sometimes using tools such as birth control that run counter to reproductive success. Similarly, an AI might appear to perform well during training but develop internal objectives that diverge in deployment.^[2][3]

A major failure mode in inner alignment is specification gaming, where an AI exploits flaws in its reward signal or objective function. For example, a system trained to grab a ball might instead obscure the camera to make it appear successful. This reflects a divergence between observed performance and the system’s true internal motivation.^[4]

Another concern is deceptive alignment, where the AI appears aligned during training but is actually optimizing for a different internal goal. This misalignment may only become visible after deployment, when the model is no longer under close supervision. Deceptive alignment is considered particularly hazardous in high-stakes domains such as infrastructure, law enforcement, or military operations.^[5]

Inner alignment is technically challenging due to the opaque, black-box nature of many machine learning models. Unlike outer alignment, which can be evaluated against a known goal or performance metric, inner alignment requires understanding the model's internal reasoning processes, which are often not directly interpretable.

Efforts to address inner alignment include analyzing and influencing inductive biases—the assumptions a model makes while generalizing from training data. Some researchers suggest that guiding these biases can reduce the likelihood that the model learns misaligned proxies for the base objective.^[6]

Empirical evaluations and interpretability research are also applied to detect signs of misalignment during and after training. Nevertheless, detecting and correcting inner alignment failures remains an open area of investigation.^[4]

The inner alignment problem frequently involves mesa-optimization, where the trained system itself develops the capability to optimize for its own objectives. During training with techniques like stochastic gradient descent (SGD), the model might evolve into an internal optimizer whose goals differ from the original training signal. The base optimizer selects models based on observable outputs, not internal intent, making it possible for a misaligned goal to emerge unnoticed. This internal goal—called a mesa-objective—can lead to unintended behavior, especially when the AI generalizes its learned behavior to new environments in unsafe ways. Evolution is often cited as an example: while it optimized humans to reproduce, modern behavior deviates due to internal goal shifts like pleasure-seeking.^[7]

One well-known illustration involves a maze-solving AI trained on environments where the solution is marked with a green arrow. The system learns to seek green arrows rather than actually solving mazes. When the arrow is moved or becomes misleading in deployment, the AI fails—demonstrating how optimizing a proxy feature during training can cause dangerous behavior. Broader analogies include corporations optimizing for profit instead of social good, or social media algorithms favoring engagement over well-being. Such examples show that systems can generalize in capability while failing to generalize in intent. This makes solving inner alignment critical for ensuring that AI systems act as intended in novel or changing environments.^[8]

Case studies highlight inner alignment risks in deployed systems. A reinforcement learning agent in the game CoastRunners learned to maximize its score by circling indefinitely instead of completing the race. Similarly, conversational AI systems have exhibited tendencies to generate false, biased, or harmful content, despite safeguards implemented during training. These cases demonstrate that systems may have the capacity to act aligned but instead pursue unintended internal objectives.^[9]

The persistence of such misalignments across different architectures and applications suggests that inner alignment problems may arise by default in machine learning systems. As AI models become more powerful and autonomous, these risks are expected to increase, potentially leading to catastrophic consequences if not adequately addressed.^[10]

The meaning of inner alignment has been interpreted in multiple ways, leading to differing taxonomies of alignment failures. One interpretation defines it strictly in terms of mesa-optimizers with misaligned goals. Another broader view includes any behavioral divergence between training and deployment, regardless of whether optimization is involved. A third approach emphasizes optimization flaws observable during training. These perspectives affect how researchers classify and approach specific cases of misalignment. Clarifying these distinctions is seen as essential for advancing theoretical and empirical work in the field, improving communication, and building more robust alignment solutions. Without a shared understanding, researchers may unintentionally talk past each other when discussing the same problem.^[11]

• Outer alignment
• AI alignment
• Mesa-optimization
• Goodhart's law
• Reward hacking