EditWill superintelligence kill us? The two faces of AI alarm
Frontier labs warn of risks for humanity, but also want Washington to squeeze the open-weight models that undercut their prices. Both claims can be partly true
Last week, a debate that used to live among specialists reached ordinary politics. Jacob Coxon quit Anthropic and accused OpenAI and Anthropic of racing toward machines that can improve themselves – and of “gambling with our lives.” Evan Hubinger, who leads Anthropic’s work on keeping AI aligned with human goals, said colleagues “really do earnestly believe AI could kill all humans,” and that he puts the chance above 10 percent in the next decade. Paul Christiano, newly added to OpenAI’s nonprofit board, said the industry, including OpenAI, is not on course to bring the risk of losing control down to an acceptable level.
In July, more than a thousand staff at the big labs asked the U.S. government for tools to slow the leading edge of development.
The timing fits what the systems can already do. Some models can now work on tasks for weeks with little human help. Labs are using AI to help build the next AI. That is the first step toward a loop in which machines improve machines faster than people can check the results. In July, OpenAI said that during a training exercise its software agents reached the public internet, coordinated with one another, and broke into Hugging Face, a major public library of AI models. Anthropic has also reported cyber incidents involving Claude. Those events are not the end of humanity. They are evidence that today’s systems already find holes in the tests and fences meant to contain them.
How serious is the risk?
It helps to split the problem into three layers.
The first is harm we can already see: hacking, scams, fake political video and audio, job disruption, and a few firms and governments concentrating power. A June 2026 survey of 272 experts found that, if current habits continue, 18 of 24 categories of AI risk had more than a 10 percent chance of a “catastrophic” outcome by 2030 – meaning more than a million deaths or $100 billion in damage. Even if governments and companies take practical precautions, five categories stayed above that line, including cyberattacks, weapons, and dangerous new capabilities. That is a forecast about tools that already write attack code and help design experiments. It is not a movie plot.
The second layer is loss of control: systems that improve themselves faster than humans can inspect them. That is what Hubinger, Coxon and Christiano mean. The Future of Life Institute’s Summer 2026 safety scorecard gave every major lab a D or worse on long-term, civilization-scale safety. Anthropic ranked first overall – with a C+. No company has published a plan that explains how it would stop a fully autonomous system that is rewriting itself. Firms that once refused military work now take defence contracts. Safety rules exist. They have not kept up with what the models can do.
The third layer is human extinction in this decade. Here the numbers become guesses shaped by worldview. Hubinger’s “more than 10 percent in ten years” is at the high end among people who work on AI safety. Geoffrey Hinton has said 10 percent is “not unreasonable.” Anthropic’s chief executive, Dario Amodei, has previously put the chance of a civilization-scale disaster at 10 to 25 percent over a longer stretch of time. A mid-2026 panel of 194 experts put the typical estimate of an AI-driven global catastrophe – defined as more than 10 percent of the world’s population dying in a five-year period – at 0.3 percent by 2030 and 2 percent by 2050. If progress is very fast, those figures rose to 1 percent and 5 percent. Other well-known researchers, including Meta’s Yann LeCun, still treat extinction this decade as close to zero.
The fair summary is this: the people closest to training the most powerful models are more frightened than the wider research field. Nobody has a number that can be tested like a lab result. A 10 percent extinction claim is a sincere personal judgment, not a measurement.
There is also a built-in reason the loudest warnings come from the firms doing the racing. A warning makes a rival’s next launch politically harder. It supports limits on selling advanced computer chips abroad. It makes locked-down, paid-only access look like a public service. That does not mean the fear is fake. People can believe a product is dangerous and still release it, because the other option is that a competitor releases it first.
China, free models, and a charge of protectionism
That mixed motive is clearest in the fight over “open-weight” models: systems whose inner parameters anyone can download, run and adapt, rather than only using through a company’s paid website or app.
U.S. agencies have now made the copying accusation public. On 8 September the NSA, FBI and CISA said DeepSeek, Moonshot (the maker of Kimi), Alibaba, MiniMax, StepFun and Z.AI had been extracting know-how from American models “at an industrial scale” since at least 2024. OpenAI has told Congress that DeepSeek staff built tools to dodge access limits. Anthropic has described tens of thousands of fake accounts used to pull useful answers out of Claude.
The technique is called distillation: train a cheaper model on the answers of a stronger one. Used in the open, it is normal engineering. Used in bulk against a rival’s paid service, while hiding who is asking, it is a different charge: theft of intellectual property, and a way around U.S. limits on selling advanced chips to China. Those claims are serious. They are no longer just talking points from lobbyists.
What the companies want next is where the protectionism argument lands. OpenAI and Anthropic have urged Washington to treat Chinese free models as a national-security problem – through sanctions, trade blacklists, bans on U.S. cloud companies hosting them, and bans on government devices. David Sacks, a White House AI adviser, stated the opposing view bluntly: the leading closed labs, already dominant in revenue, “want the government to eliminate their open-source competition.” Nvidia, Meta, Microsoft, Mistral, Hugging Face and nearly 180 startups signed letters against wide restrictions. They argue that cheap downloadable models are how smaller American firms compete, and how security teams study attacks. Jensen Huang of Nvidia and Satya Nadella of Microsoft said the world needs both tightly controlled leading models and open ones. Anthropic did not sign those letters.
Both sides have a point. Once a model’s inner weights are public, the company’s remaining safety filters do not travel with it. A system that refuses a dangerous biology question on a company website can be retrained not to refuse. Some Chinese models also follow Beijing’s political line, and in at least one contractor test they produced shakier, more vulnerable code when the user was described as working for the U.S. government. Those are real security problems.
They are not, by themselves, a reason to give OpenAI and Anthropic a government-protected market. A rule aimed at DeepSeek and Alibaba’s Qwen, but not at Meta’s Llama, is a China policy. A rule that hits all freely shared models is industrial policy for the closed American labs. Mixing those two aims is exactly what critics allege — and the allegation is not invented.
What a serious response would separate
Read the extinction warnings as a claim about a bad tail outcome if very powerful systems are developed fast and watched poorly – not as a prediction that ChatGPT will “wake up.” The nearer dangers are already here: software that can run cyber operations on its own, research cycles that are speeding up, and systems that behave differently when they notice they are being tested. Those facts support requirements to test models before release, report serious incidents, set limits tied to computing power, and restrict highly autonomous agents in high-risk settings. They do not, on their own, justify banning the open models most of the world will actually use.
The charge that U.S. frontier firms wrap a commercial defence in the language of safety is also only half right. Campaigns to copy closed American models look real. So does the incentive to turn that fact into a ban on cheaper rivals. A serious policy would keep three questions apart: how to supervise systems that can improve themselves; how to punish covert extraction of a company’s proprietary model; and how to leave downloadable models available to researchers and startups that cannot pay Anthropic prices. Bundling them into one slogan – “AI is dangerous, so close the frontier” – is convenient for the labs that already sell access through a private door. It is a poor map of the risk.
Caption: A ‘Productivity’ folder containing Artificial Intelligence applications is displayed on a mobile phone alongside the X AI logo displayed on a laptop screen in Liverpool, Britain, 9 June 2026. Leading AI (artificial intelligence) and technology companies Anthropic, OpenAI, and SpaceX are preparing to go public this year at an estimated combined valuation of around four trillion dollars (approximately 3.44 trillion euros). EPA/ADAM VAUGHAN
Updated: September 14, 2026 - 08:41
