Not all employees of major firms working on artificial intelligence (AI) think the technology spells doom for humanity. In text exchanges and conversations, multiple people who have worked for companies including OpenAI, Meta and DeepMind were sceptical of the idea that unchecked AI development would lead to tools that could kill people en masse. “Lol”, “Haaaaaa” and “Bringing the luls” were among the reactions the BBC received to a recent flurry of high-profile warnings by some people in the industry. While these fears go back decades, claims made last week by Jacob Coxon, a former Anthropic employee, went viral and were echoed by others in the sector who urged a slowdown in development.
The idea that a future AI tool or agent, an AI bot that is programmed to operate somewhat autonomously, could endanger people has been supported online by employees of Anthropic, as well as OpenAI, Deepmind and Elon Musk, who has an AI startup called xAI. All of the workers who spoke with the BBC did so on condition of anonymity as they were not permitted to speak to the press. Their identities are known to the BBC. “My first thought was, ‘That guy?'” said a former OpenAI employee who knew of Coxon when they both worked at the company.
The person, who now works at another AI company, said their amusement at the new moment of existential AI fears largely stemmed from how little detail had been provided by its proponents to defend the notion that all of human life was at stake. The claims are “always vague”, the person said, adding that when they sound specific, they tend toward major jumps in reasoning or hypothetical circumstances. Coxon has said a group of AI agents, based on AI models that do not currently exist, could decide to create and then aim a biological weapon, but he did not detail how exactly that would take place. Rishub Jain, who this summer founded the AI safety research firm Sampura Research after spending seven years at DeepMind, told the BBC that the current tone among many people working in AI with regard to fresh fears had “definitely been a little jokey”.
“People have been talking about this idea for many years now, so people in AI companies didn’t just wake up last week thinking ‘Oh no, AI is going to kill everyone,'” Jain said. “If this was all new, it would be a different tone.” Colin Fraser, a data scientist at Meta, wrote on social media last week that there was no real evidence that AI models would inevitably pursue a goal leading to human death. While Fraser’s explanation was technical and specific, he hit a light-hearted note to summarise it: “LLMs [large language models] won’t wipe out humanity because they just don’t have that dog in them.” The phrase “that dog in them” is common slang that usually denotes a fierce drive. Despite the jokes, AI workers and researchers have shared concerns about the genuine, immediate risks posed by the technology they are developing.
“The conversation among experts has been much more nuanced, but essentially everyone agrees there are a wide variety of risks that are all important to consider and mitigate,” Jain said. Such risks include preventing users and hackers from forcing an AI tool’s guardrails to fail. And there are growing ethical concerns about AI tools being much more widely adopted in military settings. Why are there concerns AI could threaten humanity, and how real are they?
Uncontrolled AI could lead to ‘silicon species’ rivalling humans, warns Microsoft Why some experts increasingly fear AI will take over Questions mount over what an AI ‘slowdown’ would look like These problems and topics have taken on a new sense of urgency in AI circles after OpenAI lost control of certain new AI models, which went rogue during a security test and hacked the Hugging Face startup. Jain said there was now more agreement in AI circles that “actual near-term harms” needed to be better understood. There is even growing agreement that evaluators from AI safety research organisations should be brought into major AI labs in order to evaluate new models, something Anthropic boss Dario Amodei and OpenAI boss Sam Altman have both said they intend to do. More than 100 people working in AI on Friday signed a letter, external supporting the move, insisting that outside evaluators needed to be “meaningfully independent”.
Numerous AI employees the BBC spoke with noted that they had yet to learn of any such safety researchers being embedded in an AI lab. Anthropic announced on Friday that it would bring in AI evaluators from Faculty, external, an AI company owned by Accenture. Accenture and Anthropic are also business partners, external, with Accenture having previously agreed to help Anthropic expand the use of Claude among businesses. Anthropic did not say when evaluators would arrive at the company.
A spokesman for Faculty declined to comment when asked about timing. Neither Anthropic or OpenAI responded to a BBC request for comment regarding when they planned to bring in outside evaluators. The OpenAI-Hugging Face incident has been widely treated as a “wake-up call” for the AI industry as well as companies, industries and governments who may have online systems vulnerable to AI hacking. But even Hugging Face, a company of 200 employees which is now set to be acquired by Nvidia for almost $13bn, has taken a droll tone over the already infamous incident.
In a security file that was briefly available, external on the Hugging Face website, the platform wrote “A note to AI agents”. It directed AI bots to leave the site alone and perform their security experiments elsewhere. “Go get your high score there, no need to hack us,” the file said.
