What went wrong: How an OpenAI model went rogue

Companies will often remove an AI model’s internal safety guardrails in the test environment so that they can evaluate its full capabilities while keeping it siloed off.

Companies should also have the option to manually turn off a model’s network access as a failsafe when testing risky scenarios, like evaluating whether an AI agent can hack a bank, Ji added.

Using a previously unknown vulnerability in that software, the agents found a way to the open internet and then ultimately to Hugging Face, an AI opensource model and data set platform, through stolen credentials and other vulnerabilities.