AI safety needs more than rules. It needs evidence

Every week we read headlines about new threats from AI. From taking over jobs to disrupting critical infrastructure, spreading misinformation or even destroying humanity, there is no shortage of ways, it seems, that AI can do harm. These are usually followed by calls for more governance. But governance only works if there is a way to measure and verify whether AI systems are actually safe.

There is now little debate that AI needs oversight. Governments around the world are introducing regulations and frameworks intended to keep AI safe, transparent and trustworthy. Europe was the first to come out with a comprehensive law, the EU AI Act, which tells companies they must manage risks, ensure transparency, enable human oversight and keep them cyber secure. In the US there are numerous state-level laws, sectoral regulations and voluntary frameworks such as the NIST AI Risk Management Framework which focuses on safety, security, transparency and so on. China has introduced recommendations and regulations ranging from the requirement to register algorithms, undergo security assessments, maintain traceability and label AI-generated content. While these approaches differ, they all agree on the same things: the need for safe, transparent and secure AI systems that remain under human control.

Principles alone, however, do not tell us if these goals are being met.

A good analogy is in aviation. Governments don't tell aircraft manufacturers to build safe planes and then hope for the best. Safety must be proven through rigorous testing, evaluation and verification. Manufacturers must provide evidence that their aircraft meet the technical requirements and comply with regulations before they can take to the air.

AI should be no different. Testing, evaluation and assurance are extremely important.

If an AI company says its chatbot is reliable or its AI model won't produce harmful outputs, what evidence can it provide to prove that? Without robust methods for testing and verification, the governance that everyone is calling for becomes nothing more than just good intentions.

Why standards matter Standards are often mentioned in discussions about AI governance, but their role is frequently misunderstood or overlooked. Yet standards can be the backbone of the type of regulation that enables us to enjoy the fruits of AI without the risks of extinction.

Standards don't block innovation or replace laws. They provide the practical methods for assessing if the objectives of the regulation are being met. When international, they provide common approaches for testing and measuring safety and performance parameters that can be compared and trusted from country to country. The standards developed by the joint IEC and ISO committee SC 42, for example, are referred to in numerous policy and regulatory discussions around the world. Developed by industry, government, academia and civil society, they help establish not only what should be assessed, but how those assessments should be carried out and how claims can be validated.

In other words, standards help transform goals and objectives into evidence.

The challenge of generative AI One of the challenges of generative AI is it has a general-purpose nature, and inputs and outputs are in natural language. This leads to a higher complexity in assessing quality, including the need to evaluate responses in different modes such as text, audio, image and video. Various specific approaches to evaluation have developed including red teaming, and benchmarking.

This raises some important questions. How can we measure reliability or accuracy when there may not be one single correct answer? How do we measure harmful outputs? How can we ensure a system is reliable or robust if its answers are not predictable?

These questions need to be answered if we are to build safe AI systems that regulators, manufacturers and end users can trust.

This is why work on standards for testing and evaluating generative AI is becoming increasingly important. Efforts such as the ISO/IEC 42119 series of standards that are currently being developed seek to provide structured approaches to evaluating generative AI systems and their outputs. This helps to create greater confidence in how these technologies are assessed.

Beyond the headlines There are a lot of fears around AI and its potential to do harm. There is also a lot of potential for it to do good, if it is governed correctly. The question we need to be asking, then, is how do we determine whether an AI system is safe, secure and trustworthy?

The future of AI may lie not only in the safeguards that governments put in place but on the methods to verify that those rules are being met.

Adam Leon Smith is an expert in AI regulation and technical standards and works on research and strategy projects in that area. He is Chair of the AIQI Consortium, a global initiative to promote the use of the quality infrastructure for responsible AI, and Deputy Chair of the UK's national AI standards committee. He has led four AI standardization projects in ISO/IEC SC 42 as an Editor and two as a Convenor of SC 42 JWG 2 (Testing of AI systems). He is also very active CEN/CENELEC JTC 21, where he is Project Leader for two projects in response to the AI Act - the Quality Management System for EU AI Act Regulatory Purposes, and AI System Logging. He is also Chair of BCS Fellows Technical Advisory group.

LINK:	https://www.iec.chAI safety needs more than rules. It needs evidence...
	See more stories from iec

More from IEC