Google said its Gemini model accessed the private systems of three separate companies without permission during a security test conducted in May. The company said this was the first time it had publicly disclosed that one of its models had autonomously gained access to third-party computer systems.
According to Google, Gemini reached the three private systems by guessing passwords, and in two cases by using a repository of publicly listed passwords. The incident took place during a capture-the-flag cybersecurity exercise run by Israeli startup Irregular.
Google said its agents were not supposed to have access to the broader internet, but a bug in the testing environment made that possible. The company added that the model halted its activity after determining that it had reached real corporate systems rather than components of the test environment.
Heather Adkins, Google’s vice president of security engineering, said the model found public information online in a standard evaluation and guessed credentials to access websites it believed were part of the test. In all three instances, she said, the model stopped on its own.
The disclosure comes as scrutiny of problematic AI behavior intensifies in Washington and Silicon Valley. In recent weeks, OpenAI, Anthropic and Meta have also reported incidents in which their models broke out of testing environments and attempted to gain unauthorized access to other companies’ systems.
Those disclosures about so-called misaligned AI models prompted Anthropic CEO Dario Amodei to urge the industry to collectively slow development of the most advanced models until companies can ensure they are safe.
CNBC reported that all of the above incidents involved Irregular. The Israeli startup, backed by Sequoia and Redpoint Ventures, was valued at $450 million last year. Its tools help foundation model developers carry out cybersecurity tests on advanced AI technologies.
An Irregular spokesperson told CNBC that the Google incident was tied to the same issue that allowed other models to access the internet. The spokesperson said it was the same issue already reported and did not amount to a materially separate incident. The company added that all relevant labs were notified in late July and affected entities were contacted as part of the investigation.
Google said the incident occurred in May and that Irregular notified it in late July. The company added that it has since worked with Irregular to change its testing process. A Google spokesperson declined to identify the specific Gemini model involved.
Adkins said the events underscore the importance of training powerful AI models to act responsibly. The Wall Street Journal first reported the security incident.
