News Items is as invaluable a way to start the day as a strong cup of coffee or a sip of mild gin" — Graydon Carter, editor and author of ‘When The Going Was Good’.
1. Dario Amodei, CEO, Anthropic:
(Over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.
My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.
My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.
I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. (Sources: darioamodei.com, openai.com, anthropic.com, metr.org, pacingthefrontier.com)
2. Steven Witt:
Even before the report by Redwood and Model Evaluation and Threat Research was published, an open letter titled “Pacing the Frontier” had begun to circulate among the leading A.I. labs. This letter called on the U.S. government to build an international regulatory body for A.I. — or, in the language of Silicon Valley, to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated A.I. development.” More than 1,000 employees signed it, including the chief scientists of OpenAI and Meta AI, and Dario Amodei, the chief executive of Anthropic. Even OpenAI’s official social media accounts endorsed it.
I have covered Silicon Valley for some time now, and I can tell you this: The existence of this letter is just as astonishing as the hacks. Both Anthropic, the developer of Claude, and OpenAI, the developer of ChatGPT, are preparing trillion-dollar I.P.O.s. Here are two companies, both posting blockbuster results, that are asking — begging, really — for the U.S. government to please come and regulate them as soon as possible. We have never seen anything like it. (Sources: penguinrandomhouse.com, pacingthefrontier.com, x.com, nytimes.com)
3. The leaders of three of the biggest AI companies agreed that they needed to slow development of the technology before their advances create a menace that can’t be controlled. It was a rare display of unity by Elon Musk, Sam Altman and Dario Amodei, who have been spending tens of billions of dollars to develop evermore powerful artificial intelligence models. Altman even suggested that his company, OpenAI, may need to delay its much-anticipated IPO to focus on safety. (Source: wsj.com)
