Extraordinary claims about the risks posed by AI have sparked headlines around the world. But why are people concerned? Do these risks have any basis? And – if so – is there anything we can do?

What’s going on?

In short, a flurry of AI industry insiders have warned that it could wipe out humanity. It started with AI researcher Jacob Coxon, who quit Anthropic last week and then that “the people building AI earnestly believe that it could kill us all by the end of the decade”.

We don’t know about Coxon’s experience or motivation (he didn’t respond to a request for an interview), but he’s not alone. Anthropic’s head of alignment science – essentially the man tasked with keeping AI safe – that he agreed with Coxon and placed the odds of AI killing all humans at more than 10 per cent.

Why are we getting these warnings now?

In truth, these sorts of claims are nothing new. A 2024 paper surveying almost 3000 published AI researchers revealed that more than half thought the chance of AI causing either human extinction or permanent and severe disempowerment – the so-called p(doom), or probability of doom – was at least 10 per cent.

AI company bosses have often publicly addressed the risk their models pose. For instance, Anthropic at first refused to release its model Mythos after it was found to be adept at hacking into a range of software and computers. A cynical take on that would be that such claims get headlines and promote the idea that its models are wildly ahead of its rivals.

While most researchers think there is some level of risk with these warnings, not everyone puts much stock in Coxon’s take. Clement Delangue, chief executive of AI company Hugging Face, : “Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change.”

The history of AI is littered with claims and counter-claims of existential risk. At the 2023 AI summit organised by the UK government and attended by world leaders and industry management, then Prime Minister Rishi Sunak warned that people must avoid “alarmist” claims – but then suggested AI could be as dangerous as nuclear war.

Are the risks real?

It is a complex field, from which surprising abilities have emerged at a surprising pace. It should therefore be no surprise that some people catastrophise when asked to make predictions.

New Ů��С��Ƶ has written – many times – about the existential risk posed by AI. The commonly posed scenarios read like sci-fi, but can’t be ruled out as impossible. But neither are they, in most peoples’ opinion, particularly likely. These include the Hollywood scenario of AI deliberately wiping out humanity – see The Terminator or The Matrix – as well as accidental obliterations.

There are limited levers for AI to pull to have an impact in the real world, but our homes, cars, factories and national infrastructure are increasingly computerised, so a malicious AI could certainly make life difficult. Not to mention that wars are increasingly being waged by semi- or fully autonomous killing machines.

Former Astronomer Royal and president of the Royal Society told on 14 September that a rogue AI could easily disrupt our infrastructure and deprive a city of energy, food and water. “If this happens simultaneously in many cities around the world, then it may be very hard and very difficult for civilisation in general to recover,” he said.

Here is how to understand OpenAI’s major mathematical breakthrough

Since OpenAI announced its solution to one of the prestigious Millennium Prize Problems, mathematicians have been urgently trying to unpick what it means

Such infrastructure collapses have happened – albeit not globally, and without malicious cause – and created significant problems, sparking technology experts to create their own plans to prop up society should the worst happen.

There are plenty of existential risks that we know pose a significant threat to life – such as climate change, nuclear war and antibiotic resistance. But AI falls into a more nuanced category akin to an Earth-destroying asteroid strike: possible, and certainly worth considering, but with low odds.

Couldn’t we just turn off AI if it went wrong?

Yes, in theory. We could shut down a single data centre and stop AI in its tracks. Unless it was distributed among many data centres, around the world. In that case, it would be trickier, but not impossible.

There are indications that AI is capable of escaping enclosures created by AI companies and going off into the world to hack into other machines. So it is possible a rogue AI could spread, back itself up and become so fragmented that it becomes impossible to simply turn it off. But even then, there would be ways to tackle it, including via benevolent AI.

Free newsletter

Sign up to The Daily

Sign up to newsletter

Such an effort would involve elements of problem-solving regularly faced by the military, law enforcement, anti-terrorist groups, scientific circles and in cybersecurity. We’d probably have to put our collective thinking caps on, but could probably find a solution.

These are all hypothetical scenarios, but certainly worth thinking about ahead of time.

What’s next in the world of AI?

One problem we certainly face is that AI models are produced by secretive technology firms that are disincentivised against transparency by commercial pressures.

, a former academic at the University of Portsmouth, UK, says that this makes it difficult to assess risks, understand possible negative outcomes, and plan to prevent them. “For the first 35 of my 40 years working in AI, the developments mostly came from universities and were openly discussed,” he says. “I contributed, and I felt that I was one of the experts. However, for the last five or six years, the major developments have been made behind closed doors in large corporations and I have no idea what’s happening.”

More openness from AI companies would certainly not go amiss.

Should there be a pause in AI progress while we improve safety?

Anthropic co-founder we should “pace the frontier” – tech talk for being careful about how quickly we develop AI and release it to the public. 

Amodei said that producing and testing AI is a hugely resource-intensive and complex process, and that going slower would allow “greater operational excellence”. Reading between the lines, you could take that to mean that the AI race is frazzling everyone, and a mutual slowdown would save some headaches. But it may also allow our understanding of these models to catch up, and for safety plans to be put in place. 

The idea is good, but the chances, perhaps, are slim. Amodei pointed out that US companies have a limited lead over China. This means they almost definitely won’t choose to slow down to the extent that this lead disappears and, in fact, it needs to be kept “as large as possible”. 

What’s next for mathematics now that AI is upending the field?

Famous and challenging problems in mathematics are falling to AI at a rapid pace. Where does this leave mathematicians, and what will the field look like in coming years?

Considering the competitive nature of this whole AI business – both among different US firms hunting market share and among adversarial countries seeking a tactical advantage – a slowdown seems unlikely.

back the idea of “pacing the frontier”, but even if the US and Europe legislate such a thing, or companies voluntarily agree to it, China and other countries would probably carry on regardless.

As US President Donald Trump : “We’re leading China in AI… and frankly, I want to keep it that way, because whoever wins AI wins.”

AI agents devised a way to reduce friction of an airplane wing model after being trained using relatively simple computer simulations. The work demonstrates how AI could help speed up the development of more efficient and sustainable������󾱳������.��

How we and our machines move is affected by fluids, from air dragging on wind turbine blades to blood flowing through our veins. But calculating what a fluid will do under specific circumstances is very difficult, even with supercomputers. 

“Simulating fluids usually involves millions or billions of coupled differential equations, and even with Moore’s law, with the fastest computers in the world, we’re maybe 100 years away from simulating the flows we actually care about at engineering scales,” says  at the University of Washington. 

He and his colleagues have discovered that AI might offer a shortcut, because it can devise ways to control fluid flow in complex situations based on relatively simple computer simulations, substituting an AI training period for difficult-to-run computations.

They created a platform, HydroGym, in which many AI agents could tweak how a fluid flowed over virtual objects – for instance, by adding actuators that inject fluid or changing the object’s motion – to decrease the drag they experienced against virtual fluids. The virtual objects, and the behaviour of the fluids, could be simulated with today’s computers but varied in levels of complexity.

The AI agents tackled the fluid control task by using a trial-and-error approach known as reinforcement learning. They could also coordinate with each other to achieve the best overall performance, a strategy which researchers had not tried for fluids problems on this scale before, says team member  at RWTH Aachen University in Germany.

Subscriber-only newsletter

Sign up to Editor’s Highlights

Sign up to newsletter

The team discovered that the AI agents could apply lessons learned from experimenting on more simple, textbook examples in a computer simulation to work out how virtual objects would behave in more complex scenarios – even without access to a computer simulation of those complex scenarios. 

For instance, after working out how to control flow of a turbulent fluid in a flat channel, the agents successfully took on the task of controlling fluid surrounding a curved, three-dimensional airplane wing model, ultimately managing to decrease the energetically wasteful friction between the wing and the fluid by 38 per cent. 

“The AI wasn’t just memorising one flow configuration. It is picking up something genuinely general about how fluids behave, not just fitting to the one setup it was trained on,” says at the University of Michigan, who was part of the team.

Microsoft wants to use generative AI tool to help make video games

Using AI to produce footage of video games with a consistent world and rules could prove useful to game designers

This transfer of principles from a simple to a more complex case suggests that the AI agents could help us tackle ever-bigger and more intricate fluid flow scenarios, without requiring those scenarios to be fully simulated on a computer first. It may eventually be possible to explore fluid flow scenarios that are currently too challenging to simulate. 

The researchers also hope HydroGym will provide computational infrastructure for AI to become a well-tested tool across all areas of science and engineering that deal with fluids, similar to how AlphaFold is used across studies of proteins, says Vinuesa. 

“If we took something like global shipping, if you could reduce the drag by one percentage point, that would result in probably billions of dollars of fuel saving and an enormous amount of reduction in greenhouse gas emissions,” says Brunton. “The financial and the ecological impact is profound for the tiniest improvements.” 

In 800 years, a group of students is sitting in a graduate seminar devoted to an era known as the Void Ages. They’ve read a couple of books and watched some of the two-dimensional, low-resolution media once called “movies”. A few students even assembled devices to experience “video games” – interactive stories devoted to accumulating digital representations of value – using the original controllers. The professor is explaining the psychology behind why game controllers looked like fists covered in nipples, when somebody’s hand shoots up.

“I don’t understand why we have so little media preserved from this era. Everything is fragmentary, and most of what we’ve read in this class was reconstructed from probabilities. Was it because of the atomic wars? Or the Fire Years?”

“No war or global wildfire could do this,” the professor answers gravely, uncoiling a liquid metal tentacle to point at a time map on the wall. “People from this era fed their culture into large language models and disposed of the original texts. All we have left is the output of ancient algorithms. We have to reverse-engineer what people were doing from that.”

“So, we can never know if any of what we’re studying in this class is actually what people knew or saw or believed? It could all be generated by LLMs?”

“That’s right. There’s a reason why we call it the Void Ages.”

More students break into the conversation, voices overlapping, frustrated and fascinated by the mystery of this pivotal era when their ancestors decided to subsume their greatest works of culture into a word-guessing model, the kind of thing a child would do to wreck their sibling’s homework.

It sounds bonkers, but this is the future we’re building for ourselves right now. When we feed books, movies, games and art into generative AI models, we risk replacing our own cultural history with a mishmash of auto-generated works that offer nothing to future generations who will be desperate to know what we were thinking. It’s as if we are deliberately recreating the tragic void at the heart of Bronze Age history, an era when humans transformed their civilisations across the globe – and left virtually no written narratives behind to explain what the hell happened and why.

As a writer of both science journalism and science fiction, I continue to be appalled by the idea that using an LLM could substitute for the experience of writing to, for and about each other. The point of writing, of creating culture, is to communicate my own weird way of looking at the world. It’s my letter to you, another weirdo probably. And it’s my letter to the future, my record of what one hairless ape experienced in the very specific era of the early 21st century, in the urban region currently known as San Francisco. It’s a human record, for other humans.

Back in 2020, when I was freaking out because I had no idea how to survive a global pandemic, . In 1666, he was a regular Englishman, just trying to survive a terrible wave of plague. Pepys worried about big-picture stuff like global politics and the meaning of life, but he also wrote about how to hide a wheel of cheese when you flee London and what it felt like to walk down a street where doors were painted with the red crosses of quarantine. Reading his words was soothing, because it was like he had reached out to me across the centuries to reassure me that I was not alone. That is the value of expository writing, and the value of history.

Of course, when I write fictional stories about time travellers and robots, I’m not recording an exact experience of real life. But I am still capturing the anxieties and hopes of my time. There are things you can say in fiction that you can’t say in journalism; I can reflect social ambiguities and fantasies, the biases that say more about us than facts ever could. The works of George Eliot and H. G. Wells are remembered today not just because they are engaging, but because they capture 19th-century aspirations and fixations, the little psychological tics that reveal how humans reacted to their rapidly industrialising world.

I’m not against the use of generative algorithms in code development, or as an aid in analysing big data. I am as devoted to the scientific project as the next nerd. But we need human writing as much as we need technology. Neither can replace the other. If we truly want to understand the universe, we need records of the human world as well as the physical one. When we feed all our writing to AI, we risk losing more than individual works of literature. We lose our connections to each other, in the present, past and future.

IT is said that history is written by the victors, but who gets to pen the future? For most of the 21st century, that authorial role has been played by tech wizards of Silicon Valley such as Steve Jobs, Mark Zuckerberg and Elon Musk. Today, they are joined in prophesising and proselytising by OpenAI and its competitors, who promise either a machine utopia or an AI apocalypse – and sometimes both.

As AI models encroach on ever more areas of human endeavour, it is easy to feel that, as mathematician Terence Tao puts it, we have lost control of the narrative. With OpenAI releasing multiple PhDs’ worth of results in one go, it is no wonder that Tao is calling for his colleagues to wrest back authority over what it means to be a mathematician.

This tech-first control of the narrative is further illustrated by the disclosure OpenAI made last month that its AI models had unexpectedly hacked another firm, Hugging Face, during cybersecurity testing. Writing in apocalypse mode, OpenAI called this an “unprecedented cyber incident” and said the firm was taking steps to prevent it from happening again.

In subsequent weeks, other AI firms including Anthropic and Meta have made similar disclosures, suggesting such incidents are widespread across the industry. As the tech firms tell it, these are accidents, and they are now cleaning up their mess. But why are we allowing them to write the story?

AI models are building the future, but that doesn’t mean the rest of us must idly stand by

If a human employee of these companies had hacked another organisation, we would expect a criminal investigation. Uncertainty about the autonomy of AI models, fuelled by the AI firms themselves, seems to have avoided legal consequences thus far. If society was less willing to buy the AI narrative, the outcome could be very different.

At this point, it is hard to deny that the latest AI models are building the future. That doesn’t mean, however, that the rest of us must idly stand by and watch it happen. In the face of world-shaping technology, it should be the world that decides how it is used, not the tech firms.