Lessons from the OpenAI-Hugging Face incident

Splashed across the worlds media is the story that an advanced AI model has gone rogue and hacked a startup. OpenAI, the firm behind ChatGPT, says two of its most advanced models escaped a test environment and breached Hugging Face, a company that hosts AI models and datasets. OpenAI has called it an unprecedented cyber incident, involving state-of-the-art cyber capabilities , but how worried should we be?

Before answering that question, let's look at what actually happened. OpenAI says it was assessing the cyber hacking skills of two of its models in a sandbox', a controlled environment where things can be tested without affecting the outside world. Unfortunately, the AI models found what is called a Zero Day, a previously unknown security flaw. It seems that OpenAI had kept an outside connection to be able to download new software into the sandbox. This was the path the models found to escape. The sole concern of the AI models was to ace the cyber hacking test. One or both of them decided the best way of doing that was to hack Hugging Face, where they believed they would find the results.

Hugging Faces security team spotted the intrusion and tried to use the frontier models of OpenAI and Claude to monitor the intrusion, but built-in guardrails prevented them from helping. In the end, Hugging Face had to use an open-source model to find the rogue AI models and contain them. The startup says there is no evidence that any of its models or datasets were affected. Meanwhile, OpenAI says it has fixed the vulnerability that let the models escape.

So, is there cause for alarm? There are two things we should worry about. Firstly, in case there is any doubt, we are in an era where AI moves faster than humans, which means that we need AIs to help us keep track of AIs. Secondly, it is no big deal that the models escaped from the sandbox. The real cause for concern is the lack of any guardrails on these models to stop them from misbehaving.

This is precisely the kind of situation that international standards like ISO/IEC 27001 and ISO/IEC 42001 are built for. Accidents will always happen, or as Murphy's Law states, anything that can go wrong will go wrong. International standards are about mitigation rather than iron clad prevention. They can help make sure that when something does go pear-shaped, the organization is ready for it and the damage is contained.

ISO/IEC 27001 is the international standard for information security management. It sets out the discipline that Hugging Face used to limit the harm. These include measures like keeping test and production environments properly separated, rotating credentials to ensure stolen ones expire quickly, giving systems and staff only the access they actually need, monitoring continuously for unusual behaviour, and having an incident response plan that gets a capable person involved within minutes, any day of the week.

Hugging Faces ability to detect and contain the intrusion quickly, rather than discover it months later, is exactly what the information security management system (ISMS) is designed to deliver.

ISO/IEC 42001 is the international standard for AI management systems. It provides guidance on formal oversight of how AI systems are tested, how to decide who is authorized to disable a safety feature and what extra safeguards have to be put in place to make up for it, how long that exception is allowed to last, and who is accountable when things go wrong.

Together, the two standards cover the infrastructure controls that contain a breach once it happens, as well as the governance controls that reduce the chance of a dangerous exception slipping through unmanaged in the first place.

The opinions expressed are those of the author and do not necessarily reflect the views of the IEC

LINK:	https://www.iec.chLessons from the OpenAI-Hugging Face incident...
	See more stories from iec

More from IEC