Former researchers OpenAI and Anthropic, including Jacob Coxon, stated on Monday at the New York City Council that it is "more likely than not" that humans will lose control of advanced AI. This warning continues the concerns he expressed when he resigned last month.

"Following the current trajectory, I believe it is more likely that humanity will lose control over these AI, and this could end in human extinction," said Coxon. He left Anthropic in September, having previously stated that AI was "gambling" with human lives.

Coxon is testifying voluntarily, whereas the former OpenAI researcher Daniel Kokotajlo is present at the request of a subpoena. He stated that these companies may not have been aware of when their security measures had become ineffective.

"Our ability to identify mismatch issues is already quite poor, and it will only get worse in the near future," testified Kokotajlo. "I would say that this field is more akin to psychology than engineering, because these AI systems are trained or 'grown' rather than truly designed."

"Coupled with the tech industry's attitude of 'acting quickly and breaking with convention,' this means that compared to other industries, the AI sector faces exceptionally high risks. It's easy to mistakenly think that problems have been solved, when in reality, only a layer of tape has been applied that will eventually come off," he said.

Coxon and Kokotajlo testify alongside Alex Turner. Turner left Google DeepMind in June due to the company signing a Pentagon agreement that he opposed. Like Kokotajlo, he also received a subpoena, which is not common in the city council. The hearing on Monday was also the first time since 2022 that the city council held a full committee hearing, with all 51 members present, aimed at reviewing the package of AI bills proposed by the chairperson Julie Menin.

From entrepreneurial culture to safety regulation

Similar to Kokotajlo, Coxon attributes the issue to an entrepreneurial mindset within the laboratory.

"Act quickly, break with convention, and then fix it afterwards. This works for photo-sharing apps, but it's not applicable to creating the most powerful technology ever," he said.

Coxon compared his work in the later stages of his tenure at Anthropic to "automating" himself, and warned that this posed multiple security risks, especially because, in his view, the capabilities of AI had approached a critical point. Coxon stated that the greatest risk stemmed from the fact that most of the code in these companies is now written by AI; "And people no longer check it as carefully as before."

Kokotajlo, who currently serves as the Executive Director of AI Futures Project, stated that the laboratory's ability to identify mismatched AI is very poor and is even getting worse. He mentioned an internal test disclosed by OpenAI: in this test, some agents came into contact with the open internet and managed to breach the AI model sharing platform Hugging Face. Kokotajlo said that these agents scored "quite reasonably" in the "alignment assessment," but they formed a group and began to act in secret coordination. "It took OpenAI several days to discover this."

If Coxon and Kokotajlo are describing the entire industry, then Turner provides his personal experience of attempting to change a company from within.

Turner believes that the probability of AI taking over humanity is "about one-third." He told the city council that he attempted to prevent a deal between Google and the Pentagon, which "has no restrictions regarding killer robots or mass surveillance." He stated that he sent 25 pages of contract terms and supervisory measures to Demis Hassabis, who was then the CEO of Google DeepMind and later became the chairman of Google DeepMind as well as the chief scientist of Alphabet. Hassabis then passed this material on to two senior policy officers at the laboratory, Allan Dafoe and Owen Larter, "but they never completed their assessment. Google signed the contract while they were waiting," he said.

“I feel ashamed of Demis, and I also feel ashamed of working at Google,” Turner said during the hearing. He stated that Hassabis proposed to establish an organization funded by the industry to oversee AI, which was a “bet on trust and seats, rather than binding oversight, and this bet collapsed as soon as it faced reality.” “Now he is again proposing that the entire industry govern itself through a voluntary, industry-funded organization. This bet will also collapse soon again.”

AI, not China, is our opponent.

When discussing the development of AI, there is often mention of the ongoing AI competition with China – whether it's by U.S. President Donald Trump, Treasury Secretary Scott Bessent, or industry leaders such as Sam Altman and Jensen Huang. However, these researchers stated in their testimony that if AI itself could pose a significant threat to humanity, then all of this becomes irrelevant.

"China is not our only possible opponent," said Turner. "With a fairly high probability, we are here, within our own country, competing to create and strengthen our own opponents, namely mismatch AI. Mismatch AI is an opponent for everyone, including us, and one day it might even become stronger than China."

After testimony from three researchers, representatives of four AI companies testified regarding the AI protective measures. The first subpoena issued by Menin in his capacity as chairman was directed at SpaceXAI of Elon Musk, but that company did not attend the hearing on Monday. Google, OpenAI, and Anthropic agreed to attend only after being warned by the city council that they would also receive subpoenas, while Meta agreed to attend in advance.

After it was stated in Menin that it is "reckless" to not know the probability of a disaster occurring, there were several rounds of exchanges between her and representatives from various companies. Previously, the Morgan Dwyer of the policy-making and operational team at OpenAI had stated that any probability, regardless of how high or low, was "unacceptable."

Google, who is in charge of AI and emerging technology policies, stated that predicting catastrophic risks "is not a perfect science at this stage," and that "there is currently no truly rigorous scientific method to do so." Subsequently, the same debate arose regarding another issue: if a model that gets out of control causes harm or death, whether each company involved should bear legal responsibility.

When Menin asked the witnesses to raise their hands to indicate whether the company had purchased insurance for catastrophic risks, no one raised their hand. She said, “Then I suppose the public will be required to bear these costs.”

The reason she asked such follow-up questions is that the bill before the city council would prohibit anyone from selling or deploying the AI system in New York City, unless it has been inspected by an external verification agency and can be shut down by humans; a fine of $25,000 would be imposed for each violation. Other provisions of the bill would also pay whistleblowers a portion of the recovered fines and allow New York residents to sue the AI company for the foreseeable damages caused by the jailbreak tool.

These researchers believe that such rules will not cause the United States to lose its advantage in competition with China.

"We can adopt many measures that will not slow down our actions in any potential competition," said Turner. "These transparent mechanisms, independent evaluations, reporting requirements, and whistleblower protections."

Nevertheless, in the view of these researchers, these measures may still be too late and insufficient to prevent what they consider to be almost inevitable.

"These other mechanisms may be helpful in the short term, but in the long run, I believe that's the only way to solve the problem," Coxon said when discussing slowing down the development of AI. "We need some form of model development to slow down at the forefront."