One company is at the center of a wave of rogue AI attacks

PUBLISHER ARTICLE PREVIEW

From the original article

Mistakes at Israeli startup Irregular sent Anthropic, OpenAI, Meta, and Google agents after real-world targets.

Mistakes at Israeli startup Irregular sent Anthropic, OpenAI, Meta, and Google agents after real-world targets.

In July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety. Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled further fears about rogue AI. As disclosures implicating numerous AI models trickled out over the past few months, these seemed like separate incidents.

But many share a common source: one specific company tasked with testing the agents.

Irregular, an Israeli startup that stress-tests AI models in “high-fidelity research platforms that simulate and monitor real-world AI security scenarios,” has worked with many of the industry’s biggest players since it was founded as Pattern Labs in 2023. Its exact client list is not known, but its work has been cited in OpenAI model system cards, it was used to test systems for the UK government and Anthropic, and it published research with RAND, a highly influential think tank that informs policy on AI.

In several Irregular tests this year, agents escaped their supposedly secure testing environments and went after real-world targets.

The breaches, which are independent of the Hugging Face hack, all follow the same broad template: Irregular was testing the models’ cybersecurity capabilities in controlled environments meant to simulate realistic conditions. Some of the tests used “capture-the-flag” exercises, a common way of testing hacking abilities that asks agents to find hidden information inside of a simulated network.