Mobile AR Interaction Design Patterns for Storytelling in Cultural Heritage: A Systematic Review

Andreas Nikolarakis; Panayiotis Koutsabasis

doi:10.3390/mti8060052

journal article Open Access Jun 17, 2024

Mobile AR Interaction Design Patterns for Storytelling in Cultural Heritage: A Systematic Review

Andreas Nikolarakis

Panayiotis Koutsabasis

Multimodal Technologies and Interaction Vol. 8 No. 6 pp. 52 · MDPI AG

View at Publisher Save 10.3390/mti8060052

Abstract

The recent advancements in mobile technologies have enabled the widespread adoption of augmented reality (AR) to enrich cultural heritage (CH) digital experiences. Mobile AR leverages visual recognition capabilities and sensor data to superimpose digital elements into the user’s view of their surroundings. The pervasive nature of AR serves several purposes in CH: visitor guidance, 3D reconstruction, educational experiences, and mobile location-based games. While most literature reviews on AR in CH focus on technological aspects such as tracking algorithms and software frameworks, there has been little exploration of the expressive affordances of AR for the delivery of meaningful interactions. This paper (based on the PRISMA guidelines) considers 64 selected publications, published from 2016 to 2023, that present mobile AR applications in CH, with the aim of identifying and analyzing the (mobile) AR (interaction) design patterns that have so far been discussed sporadically in the literature. We identify sixteen (16) main UX design patterns, as well as eight (8) patterns with a single occurrence in the paper corpus, that have been employed—sometimes in combination—to address recurring design problems or contexts, e.g., user navigation, representing the past, uncovering hidden elements, etc. We analyze each AR design pattern by providing a title, a working definition, principal use cases, and abstract illustrations that indicate the main concept and its workings (where applicable) and explanation with respect to examples from the paper corpus. We discuss the AR design patterns in terms of a few broader design and development concerns, including the AR recognition approach, content production and development requirements, and affordances for storytelling, as well as possible contexts and experiences, including indoor/outdoor settings, location-based experiences, mobile guides, and mobile games. We envisage that this work will thoroughly inform AR designers and developers abot the current state of the art and the possibilities and affordances of mobile AR design patterns with respect to particular CH contexts.

Topics

No keywords indexed for this article. Browse by subject →

References

89

[1]

Bayle "Putting it all together: Towards a pattern language for interaction design" ACM SIGCHI Bull. (1998) 10.1145/280571.280580

[2]

Björk, S., and Holopainen, J. (2004). Patterns in Game Design, Charles River Media, Inc.

[3]

Borchers, J.O. (2000, January 17–19). A pattern approach to interaction design. Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, New York, NY, USA. 10.1145/347642.347795

[4]

Spierling, U., and Kampa, A. (2017). An extensible system and its design constraints for location-based serious games with augmented reality. Serious Games: Third Joint International Conference, JCSG 2017, Valencia, Spain, 23–24 November 2017, Springer International Publishing. Proceedings 3.

[5]

Panou, C., Ragia, L., Dimelli, D., and Mania, K. (2018). An architecture for mobile outdoors augmented reality for cultural heritage. ISPRS Int. J. Geo-Inf., 7. 10.3390/ijgi7120463

[6]

Galatis, P., Gavalas, D., Kasapakis, V., Pantziou, G.E., and Zaroliagis, C.D. (December, January 30). Mobile Augmented Reality Guides in Cultural Heritage. Proceedings of the MobiCASE’16: Proceedings of the 8th EAI International Conference on Mobile Computing, Applications and Services, Cambridge, UK.

[7]

Sunar "3D object manipulation techniques in handheld mobile augmented reality interface: A review" IEEE Access (2019) 10.1109/access.2019.2906394

[8]

Cao "Mobile augmented reality: User interfaces, frameworks, and intelligence" ACM Comput. Surv. (2023)

[9]

Augmented Reality in Cultural Heritage: An Overview of the Last Decade of Applications

Răzvan Gabriel Boboc, Elena Băutu, Florin Gîrbacia et al.

Applied Sciences 10.3390/app12199859

[10]

Russo, M. (2021). AR in the Architecture Domain: State of the Art. Appl. Sci., 11. 10.3390/app11156800

[11]

Aliprantis "A survey of augmented reality applications in cultural heritage" Int. J. Comput. Methods Herit. Sci. (2019) 10.4018/ijcmhs.2019070107

[12]

Chatzopoulos "Mobile augmented reality survey: From where we are to where we go" IEEE Access (2017) 10.1109/access.2017.2698164

[13]

Challenor, J., and Ma, M. (2019). A review of augmented reality applications for history education and heritage visualisation. Multimodal Technol. Interact., 3. 10.3390/mti3020039

[14]

Kljun, M., Geroimenko, V., and Čopič Pucihar, K. (2020). Augmented reality in education: Current status and advancement of the field. Augmented Reality in Education: A New Technology for Teaching and Learning, Springer. 10.1007/978-3-030-42156-4_1

[15]

Gonzalez Vargas, J.C., Fabregat, R., Carrillo-Ramos, A., and Jové, T. (2020). Survey: Using augmented reality to improve learning motivation in cultural heritage studies. Appl. Sci., 10. 10.3390/app10030897

[16]

The PRISMA 2020 statement: An updated guideline for reporting systematic reviews

Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt et al.

International Journal of Surgery 2021 10.1016/j.ijsu.2021.105906

[17]

Koutsabasis, P., Partheniadis, K., Gardeli, A., Vogiatzidakis, P., Nikolakopoulou, V., Chatzigrigoriou, P., and Vosinakis, S. (2021, January 25–27). Location-Based Games for Cultural Heritage: Applying the Design Thinking Process. Proceedings of the CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter, Athens, Greece. 10.1145/3489410.3489419

[18]

Chiang "Augmented Reality Supported Learning for Cultural Heritage of Taiwan in On-site and Off-site Environments: The case of a Daxi Old Street" ACM J. Comput. Cult. Herit. (2023) 10.1145/3589231

[19]

Aitamurto, T., Boin, J.B., Chen, K., Cherif, A., and Shridhar, S. (2018). The impact of augmented reality on art engagement: Liking, impression of learning, and distraction. Virtual, Augmented and Mixed Reality: Applications in Health, Cultural Heritage, and Industry: 10th International Conference, VAMR 2018, Held as Part of HCI International 2018, Las Vegas, NV, USA, 15–20 July 2018, Springer International Publishing. Proceedings, Part II 10.

[20]

Alkhafaji, A., Cocea, M., Crellin, J., and Fallahkhair, S. (2017, January 10–12). Guidelines for designing a smart and ubiquitous learning environment with respect to cultural heritage. Proceedings of the 2017 11th International Conference on Research Challenges in Information Science (RCIS), Brighton, UK. 10.1109/rcis.2017.7956556

[21]

Lerma "Design and implementation of an augmented reality application for rock art visualization in Cova dels Cavalls (Spain)" J. Cult. Herit. (2019) 10.1016/j.culher.2019.03.014

[22]

Boboc, R.G., Duguleană, M., Voinea, G.D., Postelnicu, C.C., Popovici, D.M., and Carrozzino, M. (2019). Mobile augmented reality for cultural heritage: Following the footsteps of Ovid among different locations in Europe. Sustainability, 11. 10.3390/su11041167

[23]

Boboc, R.G., Gîrbacia, F., Duguleană, M., and Tavčar, A. (2017, January 22–24). A handheld Augmented Reality to revive a demolished Reformed Church from Braşov. Proceedings of the Virtual Reality International Conference-Laval Virtual, Laval, France. 10.1145/3110292.3110311

[24]

Bousbahi, F., and Boreggah, B. (2018, January 18–20). Mobile augmented reality adaptation through smartphone device based hybrid tracking to support cultural heritage experience. Proceedings of the 2nd International Conference on Smart Digital Environment, Rabat, Morocco. 10.1145/3289100.3289109

[25]

Liarokapis "A hybrid augmented reality guide for underwater cultural heritage sites" Pers. Ubiquitous Comput. (2020) 10.1007/s00779-019-01354-6

[26]

Chen "Design and Evaluation for Improving Lantern Culture Learning Experience with Augmented Reality" Int. J. Hum.–Comput. Interact. (2024) 10.1080/10447318.2023.2193513

[27]

Cushing "Walk1916: Exploring non-research user access to and use of digital surrogates via a mobile walking tour app" J. Doc. (2017) 10.1108/jd-03-2017-0031

[28]

Damala "The Loupe: Tangible augmented reality for learning to look at Ancient Greek art" Mediterr. Archaeol. Archaeom. (2016)

[29]

De Paolis, L.T., De Luca, V., and D’Errico, G. (2018). Augmented reality to understand the Leonardo’s Machines. Augmented Reality, Virtual Reality, and Computer Graphics: 5th International Conference, AVR 2018, Otranto, Italy, 24–27 June 2018, Springer International Publishing. Proceedings, Part II 5. 10.1007/978-3-319-95282-6

[30]

dela Cruz, D.R., Sevilla, J.S., San Gabriel, J.W.D., Cruz, A.J.P.D., and Caselis, E.J.S. (2018, January 15–17). Design and Development of Augmented Reality (AR) Mobile Application for Malolos’ Kameztizuhan (Malolos Heritage Town, Philippines). Proceedings of the 2018 IEEE Games, Entertainment, Media Conference (GEM), Galway, Ireland. 10.1109/gem.2018.8516272

[31]

Duguleana, M., Brodi, R., Girbacia, F., Postelnicu, C., Machidon, O., and Carrozzino, M. (2016). Time-travelling with mobile augmented reality: A case study on the piazza dei miracoli. Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection: 6th International Conference, EuroMed 2016, Nicosia, Cyprus, 31 October–5 November 2016, Springer International Publishing. Proceedings, Part I 6. 10.1007/978-3-319-48496-9_73

[32]

Fazio "Bringing empty rooms to life for casual visitors using an AR adventure game: Skullduggery at old government house" J. Comput. Cult. Herit. (2020) 10.1145/3418037

[33]

Fenu "Svevo tour: The design and the experimentation of an augmented reality application for engaging visitors of a literary museum" Int. J. Hum.-Comput. Stud. (2018) 10.1016/j.ijhcs.2018.01.009

[34]

Foukarakis, M., Faltakas, O., Frantzeskakis, G., Ntafotis, E., Zidianakis, E., Kontaki, E., Manoli, C., Ntoa, S., Partarakis, N., and Stephanidis, C. (2023, January 23–28). Connecting Historic Photographs with the Modern Landscape. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark. 10.1007/978-3-031-35998-9_55

[35]

Galani, S., and Vosinakis, S. (2022, January 4–7). Connecting Intangible Cultural Heritage and Architecture through Mobile Augmented Reality Narratives and Scale Models. Proceedings of the 2022 International Conference on Interactive Media, Smart Systems and Emerging Technologies (IMET), Limassol, Cyprus. 10.1109/imet54801.2022.9929676

[36]

Gao, G., Zhang, Y., Cheng, C., Bu, Y., and Shih, P.C. (2018, January 26–30). Designing to enhance student participation in campus heritage using augmented reality. Proceedings of the 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), San Francisco, CA, USA. 10.1109/digitalheritage.2018.8810077

[37]

Garro "Impact of Location, Gender and Previous Experience on User Evaluation of Augmented Reality in Cultural Heritage: The Mjällby Crucifix Case Study" Heritage (2022) 10.3390/heritage5030104

[38]

Hammady, R., Ma, M., and Powell, A. (2018). User experience of markerless augmented reality applications in cultural heritage museums:‘museumeye’as a case study. Augmented Reality, Virtual Reality, and Computer Graphics: 5th International Conference, AVR 2018, Otranto, Italy, 24–27 June 2018, Springer International Publishing. Proceedings, Part II 5. 10.1007/978-3-319-95282-6_26

[39]

Hincapie "Methodological framework for the design and development of applications for reactivation of cultural heritage: Case study cisneros marketplace at Medellin, Colombia" J. Comput. Cult. Herit. (2016) 10.1145/2827856

[40]

Ioannidi, A., Gavalas, D., and Kasapakis, V. (2017, January 3–6). Flaneur: Augmented exploration of the architectural urbanscape. Proceedings of the 2017 IEEE Symposium on Computers and Communications (ISCC), Heraklion, Greece. 10.1109/iscc.2017.8024582

[41]

Tian "PoeticAR: Reviving Traditional Poetry of the Heritage Site of Jichang Garden via Augmented Reality" Int. J. Hum.–Comput. Interact. (2023) 10.1080/10447318.2023.2176806

[42]

Kalmpourtzis, G., Ketsiakidis, G., Vrysis, L., Xi, T., Wang, X.L., and Dimoulas, C. (2021, January 21–23). Eliciting Educators’ Needs on the Design and Application of Augmented Reality Educational Board Games on Cultural Heritage: The case of CHARMap. Proceedings of the 2021 IEEE Global Engineering Education Conference (EDUCON), Vienna, Austria. 10.1109/educon46332.2021.9453877

[43]

Koo "Development of an augmented reality tour guide for a cultural heritage site" J. Comput. Cult. Herit. (2019) 10.1145/3317552

[44]

Kotsopoulos "An authoring platform for developing smart apps which elevate cultural heritage experiences: A system dynamics approach in gamification" J. Ambient. Intell. Humaniz. Comput. (2019) 10.1007/s12652-019-01505-w

[45]

Lee "Developing museum education content: AR blended learning" Int. J. Art Des. Educ. (2021) 10.1111/jade.12352

[46]

Lehto "Augmented reality gaming as a tool for subjectivizing visitor experience at cultural heritage locations—Case lights on!" J. Comput. Cult. Herit. (2020) 10.1145/3415142

[47]

Liang, H., Ge, C., Sun, Y., Li, P.U., Liang, F., and Wang, C. (2020, January 18–20). Prototype of porcelain safety display based on Augmented Reality Technology. Proceedings of the 2020 9th International Conference on Networks, Communication and Computing, Tokyo, Japan. 10.1145/3447654.3447663

[48]

Liestøl, G. (2018, January 26–30). The photo positioning puzzle: Creating engaging applications for historical photographs by combining mobile augmented reality and gamification. Proceedings of the 2018 3RD Digital Heritage International Congress (Digitalheritage) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), San Francisco, CA, USA. 10.1109/digitalheritage.2018.8810038

[49]

Luiro, E., Hannula, P., Launne, E., Mustonen, S., Westerlund, T., and Häkkilä, J. (2019, January 26–29). Exploring local history and cultural heritage through a mobile game. Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, Pisa, Italy. 10.1145/3365610.3368411

[50]

Concerns and Challenges Developing Mobile Augmented Reality Experiences for Museum Exhibitions

Diana Marques, Robert Costello

Curator: The Museum Journal 2018 10.1111/cura.12279

Showing 50 of 89 references

Metrics

29

Citations

89

References

Details

Published: Jun 17, 2024
Vol/Issue: 8(6)
Pages: 52
License: View

Authors

A

Andreas Nikolarakis

Department of Product and Systems Design Engineering, University of the Aegean, 84100 Syros, Greece

P

Panayiotis Koutsabasis

Department of Product and Systems Design Engineering, University of the Aegean, 84100 Syros, Greece

Cite This Article

Andreas Nikolarakis, Panayiotis Koutsabasis (2024). Mobile AR Interaction Design Patterns for Storytelling in Cultural Heritage: A Systematic Review. Multimodal Technologies and Interaction, 8(6), 52. https://doi.org/10.3390/mti8060052

Mobile AR Interaction Design Patterns for Storytelling in Cultural Heritage: A Systematic Review

You May Also Like