The potential of remote XR experimentation: Defining benefits and limitations through expert survey and case study

Jack Ratcliffe; Laurissa Tokarchuk

doi:10.3389/fcomp.2022.952996

journal article Open Access Dec 02, 2022

The potential of remote XR experimentation: Defining benefits and limitations through expert survey and case study

Jack Ratcliffe Laurissa Tokarchuk

Frontiers in Computer Science Vol. 4 · Frontiers Media SA

View at Publisher Save 10.3389/fcomp.2022.952996

Abstract

Experimentation using extended reality (XR) technology is predominantly conducted in-lab with a co-present researcher. Remote XR experiments, without co-present researchers, have been less common, despite the success of remote approaches for non-XR investigations. In order to understand why remote XR experiments are atypical, this article outlines the perceived limitations, as well as potential benefits, of conducting remote XR experiments, through a thematic analysis of responses to a 30-item survey of 46 XR researchers. These are synthesized into five core research questions for the XR community, and concern types of participant, recruitment processes, potential impacts of remote setup and settings, the data-capture affordances of XR hardware and how remote XR experiment development can be optimized to reduce demands on the researcher. It then explores these questions by running two experiments in a fully “encapsulated” remote XR case study, in which the recruitment and experiment processes is distributed and conducted unsupervised. It discusses the design, experiment, and results from this case study in the context of these core questions.

Topics

No keywords indexed for this article. Browse by subject →

References

80

[1]

Alexandrovsky "Examining design choices of questionnaires in VR user studies," (2020) 10.1145/3313831.3376260

[2]

Augmented reality (AR), virtual reality (VR), and mixed reality (MR) market size worldwide in 2021 and 2028 AlsopT Statista2022

[3]

Amores "Promoting relaxation using virtual reality, olfactory interfaces and wearable EEG," (2018) 10.1109/bsn.2018.8329668

[4]

Aronson "The theory of cognitive dissonance: a current perspective" Adv. Exp. Soc. Psychol (1969) 10.1016/s0065-2601(08)60075-1

[5]

Aust "Seriousness checks are useful to improve data validity in online research" Behav. Res. Methods (2013) 10.3758/s13428-012-0265-2

[6]

Bailey "Presence and memory: immersive virtual reality effects on cued recall," (2012)

[7]

Virtual Embodiment of White People in a Black Virtual Body Leads to a Sustained Reduction in Their Implicit Racial Bias

Domna Banakou, Parasuram D. Hanumanthu, Mel Slater

Frontiers in Human Neuroscience 2016 10.3389/fnhum.2016.00601

[8]

Blascovich "Social influence within immersive virtual environments," (2002) 10.1007/978-1-4471-0277-9_8

[9]

Blascovich "Immersive virtual environment technology as a methodological tool for social psychology" Psychol. Inq (2002) 10.1207/s15327965pli1302_03

[10]

Using thematic analysis in psychology

Virginia Braun, Victoria Clarke

Qualitative Research in Psychology 2006 10.1191/1478088706qp063oa

[11]

Brookes "Studying human behavior with virtual reality: the unity experiment framework" Behav. Res. Methods (2020) 10.3758/s13428-019-01242-0

[12]

Buchanan "Potential of the internet for personality research," (2000) 10.1016/b978-012099980-4/50006-x

[13]

Buhrmester "Amazon's mechanical turk: a new source of inexpensive, yet high-quality data?" Perspect. Psychol. Sci (2016) 10.1177/1745691610393980

[14]

Buhrmester "An evaluation of Amazon's mechanical turk, its rapid rise, and its effective use" Perspect. Psychol. Sci (2018) 10.1177/1745691617706516

[15]

Casler "Separate but equal? A comparison of participants and data gathered via Amazon's mturk, social media, and face-to-face behavioral testing" Comput. Hum. Behav (2013) 10.1016/j.chb.2013.05.009

[16]

Choi "Multisensory integration in the virtual hand illusion with active movement" Biomed. Res. Int (2016) 10.1155/2016/8163098

[17]

Clark "Research and theory on multi-media learning effects," (1992) 10.1007/978-3-642-77705-9_2

[18]

Connelly "A pneumatic glove and immersive virtual reality environment for hand rehabilitative training after stroke" IEEE Trans. Neural Syst. Rehabil. Eng (2010) 10.1109/tnsre.2010.2047588

[19]

Creasy "Effects of supervised and unsupervised physical activity programmes for weight loss" Obesity Sci. Pract (2017) 10.1002/osp4.107

[20]

Egan "An evaluation of heart rate and electrodermal activity as an objective qoe evaluation method for immersive virtual reality environments," (2016) 10.1109/qomex.2016.7498964

[21]

Ely (1994)

[22]

Freeman "Automated psychological therapy using immersive virtual reality for treatment of fear of heights: a single-blind, parallel-group, randomised controlled trial" Lancet Psychiatry (2018) 10.1016/s2215-0366(18)30226-8

[23]

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Joseph K. Goodman, Cynthia E. Cryder, Amar Cheema

Journal of Behavioral Decision Making 2013 10.1002/bdm.1753

[24]

Gosling "Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires" Am. Psychol (2004) 10.1037/0003-066x.59.2.93

[25]

Gromala "The virtual meditative walk: virtual reality therapy for chronic pain management," (2015) 10.1145/2702123.2702344

[26]

Grysman "Collecting narrative data on amazon's mechanical turk" Appl. Cogn. Psychol (2015) 10.1002/acp.3140

[27]

Henrich "Most people are not weird" Nature (2010) 10.1038/466029a

[28]

Hertzum "What do thinking-aloud participants say? A comparison of moderated and unmoderated usability sessions" Int. J. Hum. Comput. Interact (2015) 10.1080/10447318.2015.1065691

[29]

Hoffman "Immersive virtual reality for reducing experimental ischemic pain" Int. J. Hum. Comput. Interact (2003) 10.1207/s15327590ijhc1503_10

[30]

Hoyt "Social inhibition in immersive virtual environments" Presence Teleoperat. Virtual Environ (2003) 10.1162/105474603321640932

[31]

Huber "Conducting online virtual environment experiments with uncompensated, unsupervised samples" PLoS ONE (2020) 10.1371/journal.pone.0227629

[32]

Josman "Busworld: designing a virtual environment for post-traumatic stress disorder in Israel: a protocol" Cyberpsychol. Behav (2006) 10.1089/cpb.2006.9.241

[33]

Kehl "Real-time pointing gesture recognition for an immersive environment," (2004) 10.1109/afgr.2004.1301595

[34]

Kettunen "Effects of unsupervised participation over the internet on a usability study about map animation," (2018)

[35]

Keusch "Why do people participate in web surveys? Applying survey participation theory to internet survey data collection" Manage. Rev. Q (2015) 10.1007/s11301-014-0111-y

[36]

Kilteni "Drumming in immersive virtual reality: the body shapes the way we play" IEEE Trans. Visual. Comput. Graph (2013) 10.1109/tvcg.2013.29

[37]

Kilteni "The sense of embodiment in virtual reality" Presence Teleoperat. Virtual Environ (2012) 10.1162/pres_a_00124

[38]

Kourtesis "Guidelines for the development of immersive virtual reality software for cognitive neuroscience and neuropsychology: the development of virtual reality everyday assessment lab (VR-EAL), a neuropsychological test battery in immersive virtual reality" Front. Comput. Sci (2020) 10.3389/fcomp.2019.00012

[39]

Krantz "Validity of web-based psychological research," (2000) 10.1016/b978-012099980-4/50003-4

[40]

Lacroix "Effects of a supervised versus an unsupervised combined balance and strength training program on balance and muscle power in healthy older adults: a randomized controlled trial" Gerontology (2015) 10.1159/000442087

[41]

Ludlow "Virtual reality: Emerging applications and future directions" Rural Spcl. Educ. Q (2015) 10.1177/875687051503400302

[42]

Ma "Web-based VR experiments powered by the crowd," (2018)

[43]

Makransky "Adding immersive virtual reality to a science lab simulation causes more presence but less learning" Learn. Instruct (2019) 10.1016/j.learninstruc.2017.12.007

[44]

Mason "Conducting behavioral research on Amazon's mechanical turk" Behav. Res. Methods (2012) 10.3758/s13428-011-0124-6

[45]

Moon "The effects of distance in local versus remote human-computer interaction," (1998)

[46]

The Uncanny Valley [From the Field]

Masahiro Mori, Karl MacDorman, Norri Kageki

IEEE Robotics & Automation Magazine 2012 10.1109/mra.2012.2192811

[47]

Moss "Characteristics of head-mounted displays and their effects on simulator sickness" Hum. Factors (2011) 10.1177/0018720811405196

[48]

Mottelson "Virtual reality studies outside the laboratory," (2017) 10.1145/3139131.3139141

[49]

Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape

Xueni Pan, Antonia F. de C. Hamilton

British Journal of Psychology 2018 10.1111/bjop.12290

[50]

Paolacci "Inside the turk: understanding mechanical turk as a participant pool" Curr. Direct. Psychol. Sci (2014) 10.1177/0963721414531598

Showing 50 of 80 references

Metrics

2

Citations

80

References

Details

Published: Dec 02, 2022
Vol/Issue: 4
License: View

Authors

Funding

Engineering and Physical Sciences Research Council

Arts and Humanities Research Council

Cite This Article

Jack Ratcliffe, Laurissa Tokarchuk (2022). The potential of remote XR experimentation: Defining benefits and limitations through expert survey and case study. Frontiers in Computer Science, 4. https://doi.org/10.3389/fcomp.2022.952996

The potential of remote XR experimentation: Defining benefits and limitations through expert survey and case study

You May Also Like