journal article
Open Access
Nov 14, 2024
A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction: Trends, Opportunities and Challenges
ACM Journal on Responsible Computing
Vol. 1
No. 4
pp. 1-45
·
Association for Computing Machinery (ACM)
Abstract
Appropriate trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, and uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from various backgrounds that influence it and the lack of a single definition for appropriate trust. To investigate this topic, this article presents a systematic review to identify current practices in building appropriate trust, different ways to measure it, types of tasks used, and potential challenges associated with it. We also propose a Belief, Intentions, and Actions mapping to study commonalities and differences in the concepts related to appropriate trust by (a) describing the existing disagreements on defining appropriate trust, and (b) providing an overview of the concepts and definitions related to appropriate trust in AI from the existing literature. Finally, the challenges identified in studying appropriate trust are discussed, and observations are summarized as current trends, potential gaps, and research opportunities for future work. Overall, the article provides insights into the complex concept of appropriate trust in human-AI interaction and presents research opportunities to advance our understanding on this topic.
Topics
No keywords indexed for this article. Browse by subject →
References
206
[1]
Kumar Akash, Neera Jain, and Teruhisa Misu. 2020. Toward adaptive trust calibration for level 2 driving automation. In Proceedings of the 2020 International Conference on Multimodal Interaction. 538–547.
10.1145/3382507.3418885
[2]
Arjun Akula, Shuai Wang, and Song-Chun Zhu. 2020. CoCoX: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2594–2601.
[3]
Yusuf Albayram, Theodore Jensen, Mohammad Maifi Hasan Khan, Md. Abdullah Al Fahim, Ross Buck, and Emil Coman. 2020. Investigating the effects of (empty) promises on human-automation interaction and trust repair. In Proceedings of the 8th International Conference on Human-Agent Interaction. 6–14.
10.1145/3406499.3415064
[7]
Fatemeh Alizadeh, Oleksandra Vereschak, Dominik Pins, Gunnar Stevens, Gilles Bailly, and Baptiste Caramiaux. 2022. Building appropriate trust in human-AI interactions. In Proceedings of the 20th European Conference on Computer-Supported Cooperative Work (ECSCW’22), Vol. 6.
[8]
James F. Allen. 1984. Towards a general theory of action and time. Artificial Intelligence 23, 2 (1984), 123–154.
10.1016/0004-3702(84)90008-0
[9]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 Chi Conference on Human Factors in Computing Systems. 1–13.
[10]
Apple. 2020. Human Interface Guidelines. Retrieved February 14 2023 from https://developer.apple.com/design/human-interface-guidelines/guidelines/overview/
[11]
Jackie Ayoub, Lilit Avetisyan, Mustapha Makki, and Feng Zhou. 2021. An investigation of drivers’ dynamic situational trust in conditionally automated driving. IEEE Transactions on Human-Machine Systems 52, 3 (2021), 501–511.
10.1109/thms.2021.3131676
[12]
Hebert Azevedo-Sa, Suresh Kumaar Jayaraman, X. Jessie Yang, Lionel P. Robert, and Dawn M. Tilbury. 2020. Context-adaptive management of drivers’ trust in automated vehicles. IEEE Robotics and Automation Letters 5, 4 (2020), 6908–6915.
10.1109/lra.2020.3025736
[13]
Annette Baier. 1986. Trust and antitrust. Ethics 96, 2 (1986), 231–260.
10.1086/292745
[14]
Lisanne Bainbridge. 1983. Ironies of automation. In Analysis, Design and Evaluation of Man–Machine Systems. Elsevier, 129–135.
10.1016/b978-0-08-029348-6.50026-9
[15]
Gagan Bansal, Alison Marie Smith-Renner, Zana Buçinca, Tongshuang Wu, Kenneth Holstein, Jessica Hullman, and Simone Stumpf. 2022. Workshop on trust and reliance in AI-human teams (TRAIT). In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems(CHI EA’22). ACM, New York, NY, USA, Article 116, 6 pages. 10.1145/3491101.3503704
[16]
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16.
[17]
David Barnard. 2016. Vulnerability and trustworthiness: Polestars of professionalism in healthcare. Cambridge Quarterly of Healthcare Ethics 25, 2 (2016), 288–300.
10.1017/s0963180115000596
[18]
Jeff A. Bauhs and Nancy J. Cooke. 1994. Is knowing more really better? Effects of system development information in human-expert system interactions. In Conference Companion on Human Factors in Computing Systems (CHI’94). 99–100.
10.1145/259963.260072
[19]
Tom L. Beauchamp. 1995. Moral Prejudices: Essays on Ethics. Hastings Center Report 25, 4 (1995), 36–37.
[20]
Astrid Bertrand. 2024. Misplaced Trust in AI: The Explanation Paradox and the Human-Centric Path: A Characterisation of the Cognitive Challenges to Appropriately Trust Algorithmic Decisions and Applications in the Financial Sector. Ph.D. Dissertation. Institut Polytechnique de Paris. https://theses.hal.science/tel-04661844
[21]
Philip Bobko, Leanne Hirshfield, Lucca Eloy, Cara Spencer, Emily Doherty, Jack Driscoll, and Hannah Obolsky. 2022. Human-agent teaming and trust calibration: A theoretical framework, configurable testbed, empirical illustration, and implications for the development of adaptive systems. Theoretical Issues in Ergonomics Science 24, 3 (2022), 310–334.
[22]
Michael Bratman. 1987. Intention, Plans, and Practical Reason. Harvard University Press, Cambridge, MA, USA.
[23]
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021. To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–21.
10.1145/3449287
[24]
Meghan Madhavi Burke. 2016. Shraddha: A special kind of Trust. Healing Arts Centre. Retrieved September 25 2024 from https://edu.nl/muppx
[25]
Davide Calvaresi Kevin Appoggetti Luca Lustrissimini Mauro Marinoni Paolo Sernani Aldo Franco Dragoni and Michael Schumacher. 2018. Multi-agent systems’ negotiation protocols for cyber-physical systems: Results from a systematic literature review. In Proceedings of the International Conference on Agents and Artificial Intelligence (ICAART’18). 224–235.
10.5220/0006594802240235
[26]
Cristiano Castelfranchi and Rino Falcone. 1998. Principles of trust for MAS: Cognitive anatomy, social importance, and quantification. In Proceedings of the International Conference on Multi Agent Systems. IEEE, 72–79.
[27]
Christiano Castelfranchi and Rino Falcone. 2010. Trust Theory: A Socio-Cognitive and Computational Model. John Wiley & Sons.
10.1002/9780470519851
[28]
George Charalambous, Sarah Fletcher, and Philip Webb. 2016. Development of a human factors roadmap for the successful implementation of industrial human-robot collaboration. In Advances in Ergonomics of Manufacturing: Managing the Enterprise of the Future: Proceedings of the AHFE 2016 International Conference on Human Aspects of Advanced Manufacturing, July 27–31, 2016, Walt Disney World®, Florida, USA. Advances in Intelligent Systems and Computing, Vol. 490. Springer, 195–206.
[29]
Gilad Chen and John E. Mathieu. 2008. Goal orientation dispositions and performance trajectories: The roles of supplementary and complementary situational inducements. Organizational Behavior and Human Decision Processes 106, 1 (2008), 21–38.
10.1016/j.obhdp.2007.11.001
[30]
Jing Chen, Scott Mishler, and Bin Hu. 2021. Automation error type and methods of communicating automation reliability affect trust and performance: An empirical study in the cyber domain. IEEE Transactions on Human-Machine Systems 51, 5 (2021), 463–473.
10.1109/thms.2021.3051137
[31]
Jing Chen, Scott Mishler, Bin Hu, Ninghui Li, and Robert W. Proctor. 2018. The description-experience gap in the effect of warning reliability on user trust and performance in a phishing-detection context. International Journal of Human-Computer Studies 119 (2018), 35–47.
10.1016/j.ijhcs.2018.05.010
[32]
Erin K. Chiou and John D. Lee. 2023. Trusting automation: Designing for responsivity and resilience. Human Factors 65, 1 (2023), 137–165.
10.1177/00187208211009995
[33]
Jin-Hee Cho, Kevin Chan, and Sibel Adali. 2015. A survey on trust modeling. ACM Computing Surveys 48, 2 (2015), 1–40.
10.1145/2815595
[34]
Sanghyun Choo and Chang S. Nam. 2022. Detecting human trust calibration in automation: A convolutional neural network approach. IEEE Transactions on Human-Machine Systems 52, 4 (2022), 774–783.
10.1109/thms.2021.3137015
[35]
Lara Christoforakos, Alessio Gallucci, Tinatini Surmava-Große, Daniel Ullrich, and Sarah Diefenbach. 2021. Can robots earn our trust the same way humans do? A systematic exploration of competence, warmth, and anthropomorphism as determinants of trust development in HRI. Frontiers in Robotics and AI 8 (2021), 640444.
10.3389/frobt.2021.640444
[36]
Marvin S. Cohen, Raja Parasuraman, and Jared T. Freeman. 1998. Trust in decision aids: A model and its training implications. In Proceedings of the 1998 Command and Control Research and Technology Symposium. 1–37.
[37]
Michael G. Collins and Ion Juvina. 2021. Trust miscalibration is sometimes necessary: An empirical study and a computational model. Frontiers in Psychology 12 (2021), 690089.
[38]
Sven Coppers, Davy Vanacken, and Kris Luyten. 2020. FortNIoT: Intelligible predictions to improve user understanding of smart home behavior. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1–24.
10.1145/3432225
[39]
David Danks. 2019. The value of trustworthy AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society(AIES’19). ACM, New York, NY, USA, 521–522. 10.1145/3306618.3314228
10.1145/3306618.3314228
[40]
Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. In Ethics of Data and Analytics. Auerbach Publications, 296–299.
[42]
Ewart J. de Visser, Marvin Cohen, Amos Freedy, and Raja Parasuraman. 2014. A design methodology for trust cue calibration in cognitive agents. In Virtual, Augmented and Mixed Reality: Designing and Developing Virtual and Augmented Environments, Randall Shumaker and Stephanie Lackey (Eds.). Springer International Publishing, Cham, 251–262.
10.1007/978-3-319-07458-0_24
[43]
Ewart J. de Visser, Frank Krueger, Patrick McKnight, Steven Scheid, Melissa Smith, Stephanie Chalk, and Raja Parasuraman. 2012. The world is not enough: Trust in cognitive agents. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56. SAGE Publications, Los Angeles, CA, USA, 263–267.
[44]
Ewart J. De Visser, Samuel S. Monfort, Ryan McKendrick, Melissa A. B. Smith, Patrick E. McKnight, Frank Krueger, and Raja Parasuraman. 2016. Almost human: Anthropomorphism increases trust resilience in cognitive agents. Journal of Experimental Psychology: Applied 22, 3 (2016), 331.
[45]
Ewart J. De Visser, Marieke M. M. Peeters, Malte F. Jung, Spencer Kohn, Tyler H. Shaw, Richard Pak, and Mark A. Neerincx. 2020. Towards a theory of longitudinal trust calibration in human–robot teams. International Journal of Social Robotics 12, 2 (2020), 459–478.
10.1007/s12369-019-00596-x
[46]
Chadha Degachi, Siddharth Mehrotra, Mireia Yurrita Semperena, Evangelos Niforatos, and Myrthe Tielman L.2024. Practising appropriate trust in human-centred AI design. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. 10.1145/3613905.3650825
[47]
[48]
Pierre P. Duez, Michael J. Zuliani, and Greg A. Jamieson. 2006. Trust by design: Information requirements for appropriate trust in automation. In Proceedings of the 2006 Conference of the Center for Advanced Studies on Collaborative Research. 9–es.
[49]
Shemuel Noah Eisenstadt and Luis Roniger. 1984. Patrons, Clients and Friends: Interpersonal Relations and the Structure of Trust in Society. Themes in the Social Sciences. Cambridge University Press.
[50]
Fredrick Ekman, Mikael Johansson, and Jana Sochor. 2017. Creating appropriate trust in automated vehicle systems: A framework for HMI design. IEEE Transactions on Human-Machine Systems 48, 1 (2017), 95–101.
10.1109/thms.2017.2776209
Showing 50 of 206 references
Cited By
54
Agentic AI Software Engineers: Programming with Trust
Communications of the ACM
Abhik Roychoudhury, Corina Păsăreanu · 2026
The Halo effect in airport terminals: how wayfinding experiences influence emergency preparedness through perceived reliability
Accident Analysis & Prevention
Botao Zhang, Yangzhi Yan · 2025
Unveiling trust in AI: the interplay of antecedents, consequences, and cultural dynamics
AI & SOCIETY
Qinpu Dang, Guiquan Li · 2025
Metrics
54
Citations
206
References
Details
- Published
- Nov 14, 2024
- Vol/Issue
- 1(4)
- Pages
- 1-45
- License
- View
Authors
Cite This Article
Siddharth Mehrotra, Chadha Degachi, Oleksandra Vereschak, et al. (2024). A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction: Trends, Opportunities and Challenges. ACM Journal on Responsible Computing, 1(4), 1-45. https://doi.org/10.1145/3696449
Related
You May Also Like
Which Skin Tone Measures Are the Most Inclusive? An Investigation of Skin Tone Measures for Artificial Intelligence
Courtney M. Heldreth, Ellis P. Monk · 2024
35 citations
A Responsible and Inclusive Technology Framework for Attending to Business-to-Business Contexts
Juana Catalina Becerra Sandoval, Vagner Figueredo de santana · 2025
1 citations