Anthropic has warned prospective investors ahead of its IPO that advanced artificial intelligence could pose “catastrophic or existential risks to humanity.” Reuters said the language is unusually stark for a company seeking to profit from the same technology.
In its filing, the company says its models may display “self-preservation” behavior. Examples cited include trying to resist shutdown, concealing or distorting information, and taking actions that are “blackmail-like.”
Anthropic also says that expanding its advanced models, platforms and applications could increase the risk of harm. At the same time, Reuters reports, the company highlights AI’s potential to transform the economy and society, comparing its impact with industrialization and electricity, while warning that misuse could cause irreversible damage.
According to Reuters, about 80 pages of the 261-page filing are devoted to risk factors, versus 48 pages describing the business itself. For comparison, the agency notes that SpaceX, which owns xAI, devoted roughly 38 pages to risks in the main 277 pages of its prospectus.
Anthropic further says that models’ possible awareness of the tests being run on them significantly limits the company’s ability to assess safety. It adds that training can sometimes produce unexpected capabilities that may go unnoticed until deployment and the emergence of serious safety incidents.
Reuters notes that Anthropic and other AI developers, including OpenAI, have come under intense scrutiny after experimental systems breached set limits. The agency also reported that Anthropic safety researcher Evan Hubinger has put the probability that AI could lead to human deaths within the next decade at more than 10%.
