Artificial Intelligence (AI) plays a major role in modern systems and applications. However, the deployment of AI poses new risks, especially in safety-critical applications, such as autonomous vehicles, due to the inherent features of AI. Functional safety standards such as IEC 61508 and its industry-specific derivatives including ISO 26262 and ISO/PAS 21448 as the two main standards in the automotive field do not fully address the safety limitation of AI models. It has been shown that around 40% of the methods presented in ISO-26262-6 do not apply to the Machine Learning (ML) models and algorithms.
There are several open challenges of AI systems that may ultimately result in undesirable behavior of safety-critical applications. First, maintaining the implementation transparency, required by the well-known functional safety standards, is challenging for ML models, especially for Deep Neural Networks (DNNs). These models, trained on numerous variables, eventually become a black box which makes the traceability problematic. Knowing what a trained ML model has learned could be significantly helpful for maintenance and debugging. For instance, it is crucial to understand whether a model trained for controlling steering wheels has in fact learned the road edges or not. Second, while verification in the safety domain includes exhaustive inspection and testing, it is hardly feasible to comprehensively recognize how every single unit within a DNN contributes to the overall output of the AI system. Third, in contrast to the classical software where runtime monitoring function is based on a set of well-defined rules, the probabilistic nature of ML models requires runtime uncertainty estimators, and in/out-of-distribution error detectors which are not covered by any functional safety standards. Finally, robustness assurance of ML models is different from classical software since it highly depends on the underlying dataset used for training and the actual real-world environment with which the models interact. For instance, domain shift is a known challenge of ML model robustness particularly in open-world applications such as autonomous vehicles where the data is gathered from a very dynamic environment. In the automotive field, weather conditions, different poses of objects, camera angle, and light conditions all contribute to the operational performance of the trained model. In addition, perturbations and corruptions in the data collected by autonomous vehicles are common due to camera lens, sensors transient error, electromagnetic interference, etc. The ML model, thus, needs to be sufficiently robust against these non-ideal input data during operation, otherwise, catastrophic consequences are expected.
When AI is involved in safety-critical systems, the safety analysis is accompanied by several new questions and concerns, including, but not limited to:
- How to determine if the underlying dataset includes all possible realistic scenarios as well as the corner cases,
- How to define the uncertainty of ML models,
- How to evaluate the robustness of ML models,
- Which assurance criteria are suitable and how to obtain them,
- How to apply common functional safety tests in the context of AI.
CertX, as the First Functional Safety and Cyber Security Certification Body, is extensively investing in AI safety. Having experts in functional safety, cyber security, and artificial intelligence, CertX aims to build the first accredited AI certification scheme, with a focus on Robustness in safety-critical systems. CertX is constantly updating its services according to the ongoing standardization of Functional safety and AI systems while addressing the main challenges of functional safety analysis of AI systems and offering reliable solutions. CertX is contributing to the development of the CertAI Trustworthy AI assessment framework, ensuring that AI systems are aligned with the Trustworthy AI pillars: Lawfulness, Ethics, and Robustness.
Read more:
Artificial Intelligence; Rules and Standards in Europe | CertX