Trustworthy versus Explainable AI in Autonomous Vessels
4. CONCLUSIONS AND FUTURE WORK
In this paper we have argued that the topic of explainable AI needs to be unpacked in relation to both the users for whom the explanations are needed, and the different types of explanations required. We argue that explanation of processing and explanation of representation
are of different natures and play different roles. In addition, we argue that trustworthy AI in the context of autonomy is a broader concept than explainable AI. Trustworthiness depends on that the different users and the different types of explanations have been successfully matched. That is, it is not an a priori but rather a posteriori issue.
Human-machine interaction is a key feature of autonomous systems and a key topic to be addressed when exploring explainability in the context of autonomous vehicles. Interpretability and explainability methods are a part of this interaction but have so far focused on the research and development of AI. We argue there is a need for developing suitable methods for the assurance process, the end-users, and externals interacting or being affected by the AI. In this paper we have three specific users in mind: the users of the ferry, externals affected by the autonomous ferry, and assurance participants.
We have proposed different types of explanations and briefly discussed the needs for each type. Generally, end-users need real-time explanations that are interpretable and adapted to their cognitive abilities, alertness and available time to understand and act. In both an assurance context and during development, explanations should be as complete and valid as possible, and simultaneously they must be interpretable to the explainee. We argue that an understanding of the required level of interpretability and completeness of evidence is needed prior to the actual development of an autonomous system.
Explanation of processing, which maps inputs to outputs and treats the AI as a black box model, may be considered more relevant for assurance and more explainable by nature than explanation of representation, which treats the AI as a grey or white box model. However, the challenge in assurance is not the interpretability or explainability itself, but rather if the set of explanations combined can suffice as valid, complete and convincing assurance evidence. We have stated that this could be possible but will require dedicated methods and practices to be developed.
In order to be interpretable, explanations are often simplified compared to what they try to explain. This means they can be less correct or complete than the actual system and should be used with precaution in any situation requiring trust in the AI, either towards an end-user or in an assurance context.
ACKNOWLEDGEMENTS
We acknowledge that our views are the result of ongoing cooperation with NTNU on research in the fields of autonomous vessels and AI and the excellent autonomous ferry concept use-case (NTNU, 2019).
REFERENCES
DARPA. (2017, May 01). Explainable Artificial Intelligence (XAI). Retrieved August 05, 2019, from DARPA: https://www.darpa.mil/program/explainable-artificial-intelligence
DNV GL. (2018, September 01). DNVGL-CG-0264 Autonomous and remotely operated ships.
Retrieved July 15, 2019, from http://rules.dnvgl.com/docs/pdf/dnvgl/cg/2018-09/dnvgl-cg-0264.pdf
EU GDPR.ORG. (2018, January 01). Retrieved August 05, 2019, from EU GDPR.ORG:
https://eugdpr.org/
Gilpin, Bau, Yuan, Bajwa, Specter, & Kagal. (2018). Explaining Explanations: An Overview of Interpretability of Machine Learning. The 5th IEEE International Conference on Data Science and Advanced Analytics (DSAA 2018). arXiv.org. Retrieved from https://arxiv.org/abs/1806.00069
Herman, B. (2017). The Promise and Peril of Human Evaluation for Model Interpretability. NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.org. Retrieved from https://arxiv.org/abs/1711.07414
Lipton, Z. C. (2017). The Doctor Just Won't Accept That! NIPS Symposium on Interpretable ML 2017. arXiv.org. Retrieved from https://arxiv.org/abs/1711.08037
Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions.
Advances in Neural Information Processing Systems 30 (NIPS 2017). NIPS. Retrieved from https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions NTNU. (2019). Autoferry - Cross Disciplinary Research. Retrieved July 24, 2019, from NTNU -
Norwegian University of Science and Technology: https://www.ntnu.edu/autoferry
Parasuraman, R., Sheridan, T., & Wickens, C. (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 3, MAY 2000, 286-297.
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., . . . Wellman, M. (2019, April 1). Machine Behaviour. Nature, 568, 477-486.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations.
SAE International. (2018a, June 15). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. Retrieved from https://www.sae.org/standards/content/j3016_201806/
SAE International. (2018b, December 11). SAE International Releases Updated Visual Chart for Its
“Levels of Driving Automation” Standard for Self-Driving Vehicles. Retrieved from Society of Automobile Engineers (SAE) International: https://www.sae.org/news/press- room/2018/12/sae-international-releases-updated-visual-chart-for-its-%E2%80%9Clevels-of-driving-automation%E2%80%9D-standard-for-self-driving-vehicles
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM:
Visual Explanations from Deep Networks via Gradient-based Localization. 2017 IEEE International Conference on Computer Vision (ICCV).
Semcon. (2019, July 15). WHO SEES YOU WHEN THE CAR DRIVES ITSELF? Retrieved from Semcon web site: https://semcon.com/smilingcar/
Taddeo, M. (2017). Trusting Digital Technologies Correctly. Minds and Machines, 27(4), 565-568.
Retrieved from https://doi.org/10.1007/s11023-017-9450-5
Wood, M., Robbel, P., Maass, M., Tebbens, R. D., Meijs, M., Harb, M., . . . Schlicht, P. (2019).
Safety First for Automated Driving. Retrieved from https://newsroom.intel.com/wp-content/uploads/sites/11/2019/07/Intel-Safety-First-for-Automated-Driving.pdf
International Seminar on Safety and Security of Autonomous Vessels 17 - 18 September 2019, Helsinki