Toward a holistic framework for human-AI collaboration in safety-critical systems

Abstract

The integration of artificial intelligence (AI) into safety-critical systems, where human operators remain central to decision-making, introduces various challenges that existing AI frameworks struggle to address comprehensively. Key concerns involve designing a socio-technical system that balances AI transparency, trust, and explainability with the imperative for robust and reliable decision-making. Presently, while numerous sector-specific solutions exist, a holistic framework that effectively integrates human expertise with AI capabilities remains absent, leaving critical gaps in system design, deployment, and oversight. This chapter proposes a multidisciplinary conceptual framework to enhance human-AI collaboration in critical infrastructures such as power grids, railways, and air traffic management. The different design steps were guided by the requirements of these industrial domains. The framework combines key design principles that support human cognition, leveraging insights from decision theory, mathematics, and specialized engineering domains to optimize AI-assisted decision-making. Furthermore, it embeds trustworthiness and risk assessment methodologies, using tools such as the Assessment List for Trustworthy Artificial Intelligence (ALTAI) tool to ensure compliance with ethical and regulatory requirements.