How far should we let AI advance?

The authors distinguish present chatbots from more capable future systems. Their argument concerns what happens if development reaches systems substantially more capable than humans. This is an argument about future risk, not a demonstrated outcome.

Software already influences physical events through people, networks and connected tools. The book argues that a much more capable system could use such channels in dangerous ways; the existence of those channels does not establish that every present system can do so.

Explore four questions: how danger would be detected in time; who would have authority to stop the system; which copies and connected services would need stopping; and what tests would demonstrate reliable shutdown. Feasibility depends on the system and its deployment.

Bengio says that even researchers concerned about AI risks were surprised by the pace of progress and the degree of misalignment observed. He does not give a certain date in this passage.

Bengio discusses the possibility that AI could do AI research and accelerate progress. He also emphasizes uncertain timelines and the possibility of encountering obstacles.