Today’s AI won’t achieve superintelligence. But catastrophic risk is real | Sunita Verma

Many Silicon Valley leaders believe superintelligence is an imminent threat to society. The White House even published an accord about it. This isn’t enough. Geoffrey Hinton, the godfather of AI, estimates the risk of human extinction is at least 10% within the next few decades. The Anthropic CEO, Dario Amodei, argued that unchecked AI “could outrun our ability to understand and control these systems”.

In response, 1,300 leaders and employees from many leading AI companies signed a letter calling to “deliberately pace” development. Public unrest grew this summer after a series of cyberattacks by “rogue” OpenAI agents broke out of a contained environment and hacked the software company Hugging Face. Two months later, the Australian government disclosed another OpenAI agent attack had broken into the government’s health portal, stealing private health data.

To reassure the public, Jensen Huang, the chief executive of Nvidia, said on CBS that there was a “zero percent” chance AI could end humanity by 2030. Huang is right to challenge the “doomsday narratives” about human extinction. But I believe there are real risks large language models…