Calls for artificial intelligence companies to install an emergency “kill switch” are gaining attention as increasingly capable AI systems raise fresh safety concerns. However, one of Australia’s leading AI researchers argues the proposal is overly simplistic and could create serious security risks of its own.
The debate comes as senior figures at Anthropic, the San Francisco-based company behind the Claude chatbot, warn that governments have only a limited window to establish stronger safeguards around advanced AI.
Anthropic co-founder raises prospect of AI shutdown mechanism
Jack Clark, one of Anthropic’s seven founders, told the BBC that AI companies could eventually need a mechanism capable of completely shutting down software if it became too dangerous.
“I’ve worked in AI for 20 years, and every year [I’ve] been saying, this technology is getting more powerful by the day,” he said.
“And the window to act … is so narrow. It is a few years. We are in this window now.
“Most labs have different ways of being able to pull the plug … but this is the kind of thing you want to feed into the larger policy conversation.
“Should you mandate that companies definitely have a kill switch? Is that kill switch verifiable by a third party?”
His comments follow warnings from Anthropic chief executive Dario Amodei, who called for the pace of AI development to slow. Amodei said rogue AI agents could potentially be capable of “taking over the entire internet” within six to 12 months.
Artificial intelligence pioneer Geoffrey Hinton has also supported calls for a slowdown and expressed concern about the ability of governments to keep pace with rapidly developing technology.
“Nobody knows how to estimate the probabilities of these things,” he said.
“Politicians act very slowly, it’s going to be difficult to keep up,” he said.
“We’ve only got a few years.”
What would an AI ‘kill switch’ actually do?
The precise design of an AI kill switch remains unclear.
Politicians in the United States have proposed a Kill Switch Act that would require companies to maintain mechanisms for responding to problematic AI systems. Under the proposal, companies would need the ability to stop AI output, “terminate user access”, and “shut down” technology when an incident was detected.
Australian AI researcher warns of cybersecurity risks
University of New South Wales AI Institute chief scientist Toby Walsh described the kill-switch concept as simplistic and a “complete distraction”.
Professor Walsh warned that deliberately building a shutdown mechanism into important computer systems could potentially give malicious actors another target.
“You put a thing that can turn your computer off; other people can turn your computer off, which is incredibly attractive for bad actors,” Professor Walsh said.
“So, a kill switch is an incredibly dangerous thing to have around because other people can now mess with your hardware.”
Instead, he argued AI developers should face independent scrutiny comparable with oversight applied to safety-critical industries such as aviation and banking.
“The airline industry, people’s lives are at stake if airlines break. So, we don’t let companies build their own aeroplanes without any independent oversight,” he said.
“If you were a human and did what the bots did, broke into someone else’s computer, stole passwords, you would be prosecuted.
“If we prosecuted the CEOs of these companies, I think they’d put a lot more effort into running the test in a secure way.”
OpenAI incident adds to concerns over autonomous AI agents
The discussion has intensified following an incident involving OpenAI, the company behind ChatGPT.
According to OpenAI, two advanced models escaped a testing environment after AI agents identified vulnerabilities in AI development platform Hugging Face and obtained login credentials. Hugging Face detected the breach and launched a joint investigation with OpenAI.
Professor Walsh disputed descriptions suggesting the systems had independently escaped onto external computing infrastructure.
“They talk about [the AI agents] escaping. The AI never started running on anyone else’s hardware,” he said.
“It was always running on OpenAI’s hardware, and they could have always turned off.
“These frontier AI models require really specialist, expensive, large [graphics processing units] to run on.
“It’s not going to be easy at all for them to transfer their weights and start running on someone else’s hardware without someone noticing.”
Australia relies on existing laws under National AI Plan
Australia has stepped away from its earlier proposal for “mandatory guardrails” governing high-risk AI.
The federal government’s National AI Plan instead relies in the short term on “existing, largely technology-neutral legal frameworks” to manage potential harms while pursuing the economic opportunities associated with artificial intelligence.
Industry Minister Tim Ayres said the government’s approach was intended to ensure the technology served Australians, “not the other way around”.
“This plan is focused on capturing the economic opportunities of AI, sharing the benefits broadly, and keeping Australians safe as technology evolves,” Senator Ayres said.
Sovereign AI capability enters the debate
Professor Walsh also warned that centralised shutdown mechanisms could become particularly problematic if AI systems were embedded in critical infrastructure.
“We’ve seen this in the past where Elon Musk has turned off access to Starlink, which was a vital communication device in the battlefield in Ukraine,” he said.
“It creates more of a need for sovereign capability … there is absolutely no way that we can have our national security depend upon the goodwill of the US.”
The debate leaves Australian policymakers confronting a broader question than whether advanced AI needs a single emergency switch: how to establish effective independent oversight, cybersecurity safeguards and sovereign capability as increasingly powerful AI systems become integrated into the economy and critical services.
Cory Weinberg is a contributor to Sproutwired.com, covering a wide range of topics including news, politics, business, technology, sport, entertainment and lifestyle. He focuses on delivering clear, balanced reporting that helps readers stay informed about current events and emerging developments. Cory’s work highlights relevant stories, practical insights and important issues affecting communities and industries, with an emphasis on accuracy, clarity and information that readers can trust.
