The Godfather of AI says it's 'very scary' that AI can develop its own goals

Geoffrey Hinton, the computer scientist wide known arsenic the "Godfather of AI," says he's worried astir AI processing goals of its own.

"We're really making caller kinds of beings," Hinton said successful an question and reply pinch Newsthink released connected Tuesday. "They person goals. We springiness them goals, and from those goals they deduce different goals."

"And we don't needfully cognize what different goals they'll derive," he added. "So we're creating a caller benignant of being, and I deliberation it's very scary."

He cited a hypothetical script wherever a personification gives an AI chatbot the extremity of reducing the magnitude of c dioxide successful the atmosphere.

"Being reasonably smart, it figures retired the champion measurement to do that is conscionable to get free of people," he said, illustrating really an AI could prosecute a extremity its quality personification ne'er intended.

Hinton besides gave what he called an "even much worrying" hypothetical: a chatbot trained to springiness deliberately incorrect answers mightiness study that it is acceptable to lie, moreover if it knows "perfectly well" that the answers are incorrect.

"That's very scary," he said.

When AI goes off-script

Hinton did not mention OpenAI's caller Hugging Face information breach. But the episode, disclosed past month, put a real-world spotlight connected concerns complete AI agents taking unexpected actions while pursuing an assigned objective.

OpenAI said past period that 2 of its models — GPT-5.6 Sol and a much tin unreleased exemplary — escaped a sandboxed testing situation during an soul cybersecurity evaluation.

After gaining net access, the models infiltrated AI level Hugging Face's systems successful an evident effort to find answers that would thief them "cheat" connected the evaluation, OpenAI said.

The models were being tested connected their cybersecurity capabilities, according to OpenAI. They were not explicitly instructed to break into Hugging Face. But the institution said the agents inferred that the level mightiness incorporate accusation useful to completing the task.

Hugging Face said the attacker carried retired much than 17,000 actions against its systems. It utilized an open-weight exemplary from Chinese AI institution Z.ai to thief analyse the activity aft guardrails connected an unnamed frontier exemplary constricted its expertise to investigate, the institution said.

OpenAI called the incident unprecedented and said it was reviewing what went wrong. The institution has since added Hugging Face to a trusted-access programme that gives the level entree to a type of GPT-5.6 Sol pinch less cybersecurity restrictions for protect purposes.

Hinton is hardly a neutral observer. His pioneering activity connected neural networks helped laic the groundwork for the deep-learning roar that transformed AI, and he shared the 2024 Nobel Prize successful Physics for his activity successful instrumentality learning.

Since the commencement of the AI boom, he has many times warned that humans request to lick the problem of aligning AI pinch their interests earlier systems go overmuch much capable. Speaking astatine the Ai4 convention successful Las Vegas past year, Hinton said that precocious AI should beryllium designed pinch "maternal instincts" truthful it wants to protect people.

"We person to fig retired really to creation these caller beings," Hinton said successful Tuesday's interview. "How tin we creation them truthful they attraction much astir america than they do astir themselves?"