Stronger evidence: ForVerdict
Whether superintelligent AI alignment is a serious near-term risk is one of the most consequential questions in technology policy, and the literature is genuinely two-sided. The "for" side prevails on the present evidence, primarily because the alignment problem has been operationalised as a near-term technical issue applicable to systems already deployed, not only to hypothetical future superintelligences. Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané (arXiv, 2016) identified five concrete failure modes — negative side effects, reward hacking, scalable oversight, safe exploration, and distributional shift — that arise from generic features of optimising agents with imperfectly specified objectives and limited supervision. Russell (Proceedings of the 27th International Conference on Intelligent User Interfaces, 2022) extended this argument by showing formally that the standard "fixed-objective optimiser" model produces predictable misalignment regardless of capability level, and by proposing a design paradigm — provably beneficial AI built around uncertainty over human preferences — that addresses the structural cause. Bai and colleagues at Anthropic (arXiv, 2022) provided empirical evidence that techniques developed in this research programme yield concrete safety improvements in deployed large language models, demonstrating that the alignment research agenda is producing measurable engineering results, not just speculation.
On the "against" side, the critiques are real and methodologically important. Bender, Gebru, McMillan-Major, and Shmitchell (FAccT 2021) argued that the dominant superintelligence-framing of AI risk pulls funding and policy attention away from documented immediate harms — environmental cost, bias, fluent misinformation, and concentration of power — and that the framing itself is not ideologically neutral. Mitchell (arXiv 2021, "Why AI is Harder Than We Think") catalogued four common fallacies driving overconfident AI capability projections and argued that the field repeatedly mistakes narrow benchmark performance for general intelligence. Mitchell (Annals of the New York Academy of Sciences, 2021) demonstrated technically that current AI systems lack the cognitive architecture — robust abstraction and analogy-making — that general intelligence would require, undermining the assumption that scaling alone will yield superintelligence on any near-term timeline.
The synthesis: the against side correctly establishes two empirical points: superintelligence is not imminent on present technical evidence, and superintelligence-framed discourse can crowd out attention to documented near-term harms. Neither point, however, dissolves the alignment problem. The Amodei, Russell, and Bai-et-al. work argues — and partially demonstrates empirically — that the structural alignment problem is not unique to superintelligent agents but is a property of the optimisation paradigm and supervision structure used to build modern AI systems. That problem is already producing harms in deployed systems (reward hacking, specification gaming, sycophancy in language models, bias amplification), and those harms are continuous with the failure modes that would scale dangerously if capabilities continued to advance. The "for" side prevails because the question, properly read, is whether the alignment problem deserves serious technical and policy attention now, and the answer is yes — both because deployed systems already exhibit alignment-relevant failures and because, even granting Mitchell's correct point that superintelligence is not close, there is no principled reason to wait until it is. The "against" critique is best read not as evidence that alignment research is unwarranted but as a corrective demand that immediate-harm work and long-horizon alignment work must both be funded — not the former at the expense of the latter, and not the latter at the expense of the former.
Ranked by citation count.
The debate
Maya: I want to ground this in what alignment work actually produces, not in speculation about superintelligence. Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané in 2016 enumerated five concrete failure modes — reward hacking, side effects, scalable oversight, safe exploration, distributional shift — that emerge from generic features of optimisation, not from any sci-fi assumption. These are present-tense engineering problems. Bai and colleagues at Anthropic showed in 2022 with Constitutional AI that techniques developed in response to those concerns yield measurable safety improvements in deployed language models. That is alignment research producing engineering results today.
Daniel: Both of those papers are about narrow, contemporary ML systems — and that is the point. The alignment-and-superintelligence framing tries to leverage near-term technical work as evidence for speculative far-term risk, but the technical content does not establish the existential framing. Bender, Gebru, McMillan-Major, and Shmitchell argued in their 2021 FAccT paper that the dominant superintelligence framing of AI risk pulls policy and research attention away from the harms that are happening now: environmental cost, bias, fluent misinformation, concentration of power. That framing is not ideologically neutral.
Maya: Granted. But the response is to fund both — not to dismiss alignment research because some of its proponents talk about superintelligence too loudly. Russell's 2022 paper in the Proceedings of the 27th International Conference on Intelligent User Interfaces formalised the alignment problem as a property of the standard optimisation-with-fixed-objective paradigm. It does not depend on superintelligence; it depends only on systems being deployed at scale to optimise objectives we cannot fully specify. That description fits current deployed AI exactly.
Daniel: Then the right framing is "AI safety for current systems," not "superintelligent alignment." Mitchell argued in her 2021 arXiv paper "Why AI is Harder Than We Think" that the field repeatedly conflates narrow benchmark performance with general intelligence — that is one of four fallacies driving over-confident capability projections. And in the Annals of the New York Academy of Sciences in 2021, she showed that no current AI system has the cognitive architecture — robust abstraction and analogy-making — that general intelligence would require. The technical state of the art does not support claims that superintelligence is around the corner.
Maya: That is a fair empirical claim about timelines, and I am not going to argue we are five years from superintelligence. The point is the alignment problem does not need superintelligence to be a problem. Reward hacking is happening in current language models. Specification gaming is a documented behaviour in current reinforcement-learning systems. Sycophancy and reward-model exploitation are observed in current RLHF pipelines. The continuity argument is exactly the point: the failure modes scale with capability, and we should not wait until they scale to catastrophe before working on them.
Daniel: The continuity argument is rhetorically powerful but technically slippery. Reward hacking in current LLMs is a problem of imperfect training; calling that the same kind of problem as a misaligned superintelligence pursuing instrumentally convergent goals is conflating two different categories. The Bender and Mitchell critiques are pointing at exactly that conflation: present harms are real, present technical limits are real, and stretching them into existential narratives serves rhetorical purposes more than technical ones.
Maya: I accept that the most extreme x-risk framings outrun the technical evidence. But the moderate version of the for-side argument — that alignment is a near-term, deployed-system problem with continuity to long-horizon risks if capabilities scale — is supported by the Amodei, Russell, and Bai-et-al. line of work. That is also what the major industrial AI labs are spending billions on. It would be strange if the entire frontier-AI research community were misallocating resources at this scale on the basis of conflations.
Daniel: It would not be strange at all. Industrial alignment teams have strong incentives, including reputational and regulatory ones, to frame their work in terms that legitimise massive AI deployment by claiming the deployment can be made safe. The Bender paper made exactly this point: the framing serves the deployers. Spending on alignment is also a hedge — it is rational for firms even if the existential risk is small, because the reputational risk of being wrong is large.
Maya: That is true, and it does not invalidate the technical content. The question is whether alignment work produces real safety progress, not whether it is also useful for industrial reputation management. The Constitutional AI result is a measurable improvement on a measurable harm. RLHF, despite its limitations, is a measurable improvement on prior baselines. The technical contributions stand independently of the framing arguments.
Daniel: I will agree that current alignment techniques produce real safety improvements on current systems. What I will not concede is that this body of work establishes that superintelligent AI alignment is a serious near-term risk. It establishes that AI safety is a serious near-term concern, which is a different and weaker claim than the one in the topic title.
Maya: That is a sharper distinction than the public discourse usually makes, and it is a useful one. The for-side claim, properly stated, is: the alignment problem is real, present, and continuous with risks that grow as capabilities scale; it is therefore a serious near-term issue that we should work on now rather than wait for superintelligence to be imminent. Mitchell's correctly cautious timeline argument is compatible with that conclusion — it just adjusts how much of the budget should go to long-horizon scenarios versus near-term harms.
Daniel: Then we are agreeing on the policy upshot — fund near-term safety work, fund immediate-harms work, be skeptical of the most aggressive existential-risk framings — and disagreeing on the framing of the question. That is a useful place to land.
Frequently asked questions
What is the AI alignment problem?
It is the challenge of making AI systems pursue what humans actually want rather than an imperfectly specified objective. Research cited here argues that systems built to optimise fixed objectives can disregard human preferences not captured in those objectives, regardless of capability level.
Is superintelligent AI coming soon?
The evidence reviewed suggests not on present technical grounds. Mitchell's work argues current systems lack robust abstraction and analogy-making, and that the field often mistakes narrow benchmark performance for general intelligence.
Are there alignment problems in AI systems today?
Yes, according to the evidence summarised here. Deployed systems already show alignment-relevant failures such as reward hacking, specification gaming, sycophancy in language models and bias amplification.
Does focusing on AI existential risk distract from current AI harms?
Bender, Gebru and colleagues argue that superintelligence framing can pull funding and policy attention away from documented harms such as bias, misleading outputs, environmental cost and concentration of power. The overall verdict holds that both immediate-harm work and long-horizon alignment work warrant support.