Dan Milmo , The Guardian; AI godfathers warn of runaway ‘intelligence explosion’
"Their concerns focus on the possibility of AIs being able to improve themselves without human intervention, a process known as “recursive self-improvement”. In the paper it is broadly referred to as automated AI research and development. Once AI systems reach expert-level capabilities at AI R&D, a single developer could run a workforce equivalent to “millions” of top human researchers, the paper says.
The authors say the automating of AI R&D is the most likely source of an intelligence explosion, because AIs are already contributing to improving their own technology and the resulting improved systems can be rapidly deployed once built.
“An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government,” the paper says.
Urging governments to take action quickly, the authors say: “Once an intelligence explosion begins, the window for action may close.”
They recommended a trio of policy priorities: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion.
Anthropic and OpenAI have both agreed to having independent evaluators assess their models after warning that AI development was reaching a critical pitch in terms of safety."
