Training birdsong recognition using virtual reality

Abstract

Background

In mega-biodiverse environments, where different species are more likely to be heard than seen, species monitoring is generally performed using bioacoustics methodologies. Furthermore, since bird vocalizations are reasonable estimators of biodiversity, their monitoring is of great importance in the formulation of conservation policies. However, birdsong recognition is an arduous task that requires dedicated training in order to achieve mastery, which is costly in terms of time and money due to the lack of accessibility of relevant information in field trips or even specialized databases. Immersive technology based on virtual reality (VR) and spatial audio may improve species monitoring by enhancing information accessibility, interaction, and user engagement.

Methods

This study used spatial audio, a Bluetooth controller, and a head-mounted display (HMD) to conduct an immersive training experience in VR. Participants moved inside a virtual world using a Bluetooth controller, while their task was to recognize targeted birdsongs. We measured the accuracy of recognition and user engagement according to the User Engagement Scale.

Results

The experimental results revealed significantly higher engagement and accuracy for participants in the VR-based training system than in a traditional computer-based training system. All four dimensions of the user engagement scale received high ratings from the participants, suggesting that VR-based training provides a motivating and attractive environment for learning demanding tasks through appropriate design, exploiting the sensory system, and virtual reality interactivity.

Conclusions

The accuracy and engagement of the VR-based training system were significantly high when tested against traditional training. Future research will focus on developing a variety of realistic ecosystems and their associated birds to increase the information on newer bird species within the training system. Finally, the proposed VR-based training system must be tested with additional participants and for a longer duration to measure information recall and recognition mastery among users.

Keyword

Human computer interaction ; Virtual environment ; Birdsong ; Audio training ; User engagement

Cite this article

Carlos ARCE-LOPERA, María José ARIAS, Gustavo CORRALES. Training birdsong recognition using virtual reality. Virtual Reality & Intelligent Hardware, 2021, 3(5): 397-406 DOI:10.1016/j.vrih.2021.09.001

References

1. Myers N, Mittermeier R A, Mittermeier C G, da Fonseca G A, Kent J. Biodiversity hotspots for conservation priorities. Nature, 2000, 403(6772): 853–858 DOI:10.1038/35002501

2. Baptiste M P, Loaiza L M G, Acevedo-Charry O, Acosta-Galvis A R, Wong L J. Global register of introduced and invasive species- Colombia. 2018

3. Krause T. Reducing deforestation in Colombia while building peace and pursuing business as usual extractivism? Journal of Political Ecology, 2020, 27(1) DOI:10.2458/v27i1.23186

4. KvsnR R, MontgomeryJ, GargS, CharlestonM. Bioacoustics data analysis-A taxonomy, survey and open challenges. IEEE Access, 2020, 8: 57684–57708 DOI:10.1109/access.2020.2978547

5. Venier L A, Mazerolle M J, Rodgers A, McIlwrick K A, Holmes S, Thompson D. Comparison of semiautomated bird song recognition with manual detection of recorded bird song samples. Avian Conservation and Ecology, 2017, 12(2): art2 DOI:10.5751/ace-01029-120202

6. Suh A, Prophet J. The state of immersive technology research: a literature analysis. Computers in Human Behavior, 2018, 86: 77–90 DOI:10.1016/j.chb.2018.04.019

7. Alrehaili E A, Al Osman H. A virtual reality role-playing serious game for experiential learning. Interactive Learning Environments, 2019, 1–14 DOI:10.1080/10494820.2019.1703008

8. Goh D H L, Lee C S, Razikin K. Interfaces for accessing location-based information on mobile devices: an empirical evaluation. Journal of the Association for Information Science and Technology, 2016, 67(12): 2882–2896 DOI:10.1002/asi.23566

9. Munafo J, Diedrick M, Stoffregen T A. The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects. Experimental Brain Research, 2017, 235(3): 889–901 DOI:10.1007/s00221-016-4846-7

10. Ibáñez M B, Di-Serio Á, Villarán-Molina D, Delgado-Kloos C. Support for augmented reality simulation systems: the effects of scaffolding on learning outcomes and behavior patterns. IEEE Transactions on Learning Technologies, 2016, 9(1): 46–56 DOI:10.1109/tlt.2015.2445761

11. Frank J A, Kapila V. Mixed-reality learning environments: Integrating mobile interfaces with laboratory test-beds. Computers & Education, 2017, 110: 88–104 DOI:10.1016/j.compedu.2017.02.009

12. Loup-Escande E, Frenoy R, Poplimont G, Thouvenin I, Gapenne O, Megalakaki O. Contributions of mixed reality in a calligraphy learning task: Effects of supplementary visual feedback and expertise on cognitive load, user experience and gestural performance. Computers in Human Behavior, 2017, 75: 42–49 DOI:10.1016/j.chb.2017.05.006

13. Ke F F, Lee S, Xu X H. Teaching training in a mixed-reality integrated learning environment. Computers in Human Behavior, 2016, 62: 212–220 DOI:10.1016/j.chb.2016.03.094

14. Hong J, He J J, Lam B, Gupta R, Gan W S. Spatial audio for soundscape design: recording and reproduction. Applied Sciences, 2017, 7(6): 627 DOI:10.3390/app7060627

15. Berger C C, Gonzalez-Franco M, Tajadura-Jiménez A, Florencio D, Zhang Z Y. Generic HRTFs may be good enough in virtual reality. improving source localization through cross-modal plasticity. Frontiers in Neuroscience, 2018, 12: 21 DOI:10.3389/fnins.2018.00021

16. O’Brien H L, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies, 2018, 112: 28–39 DOI:10.1016/j.ijhcs.2018.01.004

17. de Winter J F C, Dodou D. Five-point Likert Items: t test versus Mann-Whitney-Wilcoxon (Addendum added October 2012). Practical Assessment, Research, and Evaluation, 2019

18. Meek G E, Ozgur C, Dunning K. Comparison of the t vs. wilcoxon signed-rank test for likert scale data and small samples. Journal of Modern Applied Statistical Methods, 2007, 6(1): 91–106 DOI:10.22237/jmasm/1177992540

19. Harpe S E. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning, 2015, 7(6): 836–850 DOI:10.1016/j.cptl.2015.08.001

20. Nguyen D, Meixner G. Gamified augmented reality training for an assembly task: a study about user engagement. In: Proceedings of the 2019 Federated Conference on Computer Science and Information Systems. IEEE, 2019 DOI:10.15439/2019f136

21. Ruan S, Jiang L W, Xu J, Tham B J K, Qiu Z N, Zhu Y S, Murnane E L, Brunskill E, Landay J A. QuizBot: a dialogue-based adaptive learning system for factual knowledge. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow Scotland Uk, New York, NY, USA, ACM, 2019 DOI:10.1145/3290605.3300587

22. Coxon M, Kelly N, Page S. Individual differences in virtual reality: Are spatial presence and spatial ability linked? Virtual Reality, 2016, 20(4): 203–212 DOI:10.1007/s10055-016-0292-x

23. Arino J J, Juan M C, Gil-Gómez J A, Mollá R. A comparative study using an autostereoscopic display with augmented and virtual reality. Behaviour & Information Technology, 2014, 33(6): 646–655 DOI:10.1080/0144929x.2013.815277

24. Kaplan A D, Cruit J, Endsley M, Beers S M, Sawyer B D, Hancock P A. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Human Factors, 2021, 63(4): 706–726 DOI:10.1177/0018720820904229

25. Huang H M, Rauch U, Liaw S S. Investigating learners' attitudes toward virtual reality learning environments: Based on a constructivist approach. Computers & Education, 2010, 55(3): 1171–1182 DOI:10.1016/j.compedu.2010.05.014

26. Huang T L, Liao S L. Creating e-shopping multisensory flow experience through augmented-reality interactive technology. Internet Research, 2017, 27(2): 449–475 DOI:10.1108/intr-11-2015-0321

27. Rutter C E, Dahlquist L M, Weiss K E. Sustained efficacy of virtual reality distraction. The Journal of Pain, 2009, 10(4): 391–397 DOI:10.1016/j.jpain.2008.09.016

2. Jun-Hao YIN, Chin-Boon CHNG, Pooi-Mun WONG, Nicholas HO, Matthew CHUA, Chee-Kong CHUI, VR and AR in human performance research―An NUS experience Virtual Reality & Intelligent Hardware 2020, 2(5): 381-393

3. Vivian GÓMEZ, Kelly PEÑARANDA, Pablo FIGUEROA, Lessons learned from requirements gathering for virtual reality simulators Virtual Reality & Intelligent Hardware 2021, 3(5): 407-422

4. Jiaxin LIU, Hongxin ZHANG, Chuankang LI, COMTIS: Customizable touchless interaction system for large screen visualization Virtual Reality & Intelligent Hardware 2020, 2(2): 162-174

5. Zixiang ZHAO, Quanwei ZHOU, Xiaoguang HAN, Lili WANG, Dynamic targets searching assistance based on virtual camera priority Virtual Reality & Intelligent Hardware 2021, 3(6): 484-500

6. Ahmed L ALYOUSIFY, Ramadhan J MSTAFA, AR-assisted children book for smart teaching and learning of Turkish alphabets Virtual Reality & Intelligent Hardware 2022, 4(3): 263-277

7. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

8. Abhishek MUKHOPADHYAY, G S Rajshekar REDDY, KamalPreet Singh SALUJA, Subhankar GHOSH, Anasol PEÑA-RIOS, Gokul GOPAL, Pradipta BISWAS, Virtual-reality-based digital twin of office spaces with social distance measurement feature Virtual Reality & Intelligent Hardware 2022, 4(1): 55-75