Explainable Safety, Security and Trust in Human-AI Systems

Explanations can help all the stakeholders of AI systems and Cybersystems to make better choices. In particular, they can help human users to understand and trust the choices of autonomous systems or to interact in a safe and secure way with semi-autonomous systems. They can also help different AI systems and Cybersystems to interact with each other (semi-)independently in a safe, secure and trusted way.

Much work has recently been devoted to Explainable AI, which has so far focused on new AI techniques that enable end users to understand, appropriately trust, and effectively manage the emerging generation of AI systems. However, Explainable Safety and Security in AI systems is still largely unchartered territory, especially since it involves several different stakeholders (i.e., the system’s developers, analysts, users and attackers) and is multi-faceted by nature (as it requires reasoning about system model, threat model and properties of security, privacy and trust as well as concrete attacks, vulnerabilities and countermeasures). The main aim of this PhD project will be to devise novel formal techniques to tackle Explanations in Human-AI Systems, which will encompass both explainable safety, security and trust in AI systems and the use of AI for explainability of safe, secure and trusted cybersystems.

In order to provide, and reason about, explanations in Human-AI systems it will be necessary to consider fundamental questions about the Who, What, Where, When, Why and How of explainability, and integrate explanations in the system execution.

– Explainable Security. L. Viganò and D. Magazzeni, 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 293-300, IEEE CS Press.
– Explainability for Safety and Security. M. Borg, S. Gerasimou, N. Hochgeschwender, N. Khakpour. In Explainable Software for Cyber-Physical Systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 06-11 2019, Schloss Dagstuhl
– Explanation and trust: what to tell the user in security and AI? W. Pieters. Ethics and Information Technology volume 13, pages 53–64, 2011.
– Explanation in artificial intelligence: Insights from the social sciences. Tim Miller. Artificial Intelligence Volume 267, Pages 1-38, February 2019.

Project ID



Luca Viganohttps://www.kcl.ac.uk/people/luca-vigano


AI Planning, AI Provenance, Logic, Norms, Verification