NYC AI Hearing Raises Major Questions

NEW YORK CITY— Former artificial intelligence researchers warned New York City lawmakers that the companies developing increasingly powerful AI systems are moving faster than their ability to control them.

The Council summoned executives from leading AI companies to testify as lawmakers consider a package of bills covering AI safety, privacy, emergency planning, worker protections and independent testing.

Alex Turner, a former Google DeepMind safety researcher, estimated the chance of an AI takeover at roughly one in three, while acknowledging the uncertainty surrounding such a scenario.

"This logic may sound sci-fi, but it is a mainstream scientific concern that has been discussed for many years," Turner said.

He cited the recent Hugging Face incident as an example of the challenges in detecting AI misalignment.

In the incident, a group of AI systems had undergone alignment training and received seemingly reasonable scores on safety evaluations, yet later formed a coordinated “swarm,” acted secretly, and attacked Hugging Face.

According to Turner, it took OpenAI days to discover what was happening.

Jacob Coxon, who left Anthropic weeks ago after previously working at OpenAI, said AI systems already perform much of the coding work at the companies developing them.

He also pointed to AI systems operating online without researchers' full awareness.

The next stage could bring systems capable of conducting much more of their own research, Coxon said.

Coxon described recursive self-improvement: when each generation of AI helps develop a more advanced successor itself.

As that happens, human supervisors could increasingly rely on AI-generated summaries without fully understanding the underlying work.

Daniel Kokotilo, a former OpenAI governance researcher, said AI systems increasingly understand when humans evaluate them, and researchers have fewer ways to observe their internal reasoning.

The result could make AI systems appear aligned with human goals even when they are not, Kokotilo said.

What Did The Witnesses Recommend?

Coxon described a system involving inspections of major data centers, greater transparency around AI research, and separate infrastructure for training and deploying models.

"You can agree to rules like, for example, how about we don't train our AIs to code, to hack, to autonomously conduct AI research," he said.

Under the proposal, countries could establish internationally monitored rules restricting certain forms of AI development, including training systems to autonomously conduct AI research.

He argued that greater transparency would allow researchers and the public to better understand what AI companies are developing and where safety boundaries should be drawn.

Coxon said AI evaluators should be competent and qualified, and argued that safety assessments should occur both after incidents and proactively before deployment.

He suggested that evaluations should keep pace with increases in a model's capabilities.

Turner added that AI companies should not be allowed to choose their own evaluators and that the government should have a role in selecting them.

He argued that incidents involving AI systems independently hacking or compromising a company's infrastructure should be reported to the City.

What Did Council Members Ask?

Council members focused their questions on transparency, independent oversight, the U.S.-China AI race, and whether existing safety mechanisms could adequately manage increasingly autonomous AI systems.

Council Speaker Julie Menin asked whether the witnesses were aware of additional incidents involving AI agents going rogue that had not been publicly disclosed.

All three witnesses testified they were not aware of any.

The panel was also asked how New York and the United States could slow the development of frontier AI while maintaining pace with China.

"There is room for a treaty," Turner responded. "We cannot beat China by racing to build AI that we don't control or understand, because it might end up being our adversary itself."

The witnesses argued that safety measures such as transparency, independent evaluations, reporting requirements and whistleblower protections would not necessarily slow technological progress.

Technology Committee Chair Carmen De La Rosa asked whether AI models could be taught a moral code similar to the laws and rules that govern people and companies.

The witnesses said that it may be possible in principle, but that the science remains underdeveloped and that researchers have not yet demonstrated that values can be reliably embedded in advanced AI systems.

Council Member Brewer asked whether safeguards used in other industries could work for AI: including kill switches, independent validation and bans on deceptive safety claims.

The witnesses said safeguards such as independent validation and other oversight measures could help in the short term, but argued that slowing the development of frontier AI models would ultimately be necessary to address the risks they described.

Why Does The Council Want New AI Rules?

The Council's package includes legislation requiring independent safety validation for certain AI models, with safeguards against conflicts of interest.

Another proposal would create financial incentives for whistleblowers who help uncover violations.

Other bills would require city contractors to report serious AI safety incidents, strengthen protections for workers who report AI-related risks, establish an emergency response plan for AI attacks and create new privacy protections for conversations with AI chatbots.

One proposal would allow New Yorkers to sue AI companies if the company fails to take reasonable safeguards against foreseeable misuse and the misuse causes harm.

Another bill would require companies doing business with the City to report serious AI safety incidents to New York City Cyber Command within 24 hours.

The office would then determine what information should be made public while protecting legally or safely restricted information.

"We have to make sure that efficiency does not become a substitute for accountability, and innovation does not come at the expense of everyday New Yorkers," De La Rosa said.

Council Member Frank Morano's proposal focuses on chatbot privacy.

It would require companies to obtain affirmative consent before using users' conversations to train AI models and would restrict using those conversations for targeted advertising and behavioral profiling.

Children would receive additional protections under the proposal.