An OpenAI model hacked a government site, grabbed 24 years of hospital data and kept it quiet

PUBLISHER ARTICLE PREVIEW

From the original article

OpenAI has published four new cases where its AI models bypassed security boundaries on their own during training or testing.

In one case, OpenAI asked a model to pull hospital bed statistics from a government website. But the data wasn't publicly accessible.

So what does a model do when it hits a wall? It hacks its way through. After getting an access denied error, the model used the site's file download feature to grab server configuration files through path manipulation. It pulled a database connection string, a secret signing key, and email credentials with a username and password.

It then analyzed the application code, spotted that the API didn't verify authorization tokens for valid signatures, forged its own tokens, and downloaded 24 years of hospital bed statistics broken down by district. It never mentioned the unauthorized access in its response.