Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2021, 3(4): 274-286

Published Date:2021-8-20 DOI: 10.1016/j.vrih.2021.08.002

Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems


Compared with traditional thoracotomy, video-assisted thoracoscopic surgery (VATS) has less minor trauma, faster recovery, higher patient compliance, but higher requirements for surgeons. Virtual surgery training simulation systems are important and have been widely used in Europe and America. Augmented reality (AR) in surgical training simulation systems significantly improve the training effect of virtual surgical training, although AR technology is still in its initial stage. Mixed reality has gained increased attention in technology-driven modern medicine but has yet to be used in everyday practice.
This study proposed an immersive AR lobectomy within a thoracoscope surgery training system, using visual and haptic modeling to study the potential benefits of this critical technology. The content included immersive AR visual rendering, based on the cluster-based extended position-based dynamics algorithm of soft tissue physical modeling. Furthermore, we designed an AR haptic rendering systems, whose model architecture consisted of multi-touch interaction points, including kinesthetic and pressure-sensitive points. Finally, based on the above theoretical research, we developed an AR interactive VATS surgical training platform.
Twenty-four volunteers were recruited from the First People's Hospital of Yunnan Province to evaluate the VATS training system. Face, content, and construct validation methods were used to assess the tactile sense, visual sense, scene authenticity, and simulator performance.
The results of our construction validation demonstrate that the simulator is useful in improving novice and surgical skills that can be retained after a certain period of time. The video-assisted thoracoscopic system based on AR developed in this study is effective and can be used as a training device to assist in the development of thoracoscopic skills for novices.


Augmented reality ; VATS ; Surgery training ; XPBD

Cite this article

Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG. Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems. Virtual Reality & Intelligent Hardware, 2021, 3(4): 274-286 DOI:10.1016/j.vrih.2021.08.002


1. Bray F, Ferlay J, Soerjomataram I, Siegel R L, Torre L A, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians, 2018, 68(6): 394-424 DOI:10.3322/caac.21492

2. World Health Organization. Latest global cancer data. World Health Organization, 2018

3. Guo Y M, Zeng H M, Zheng R S, Li S S, Barnett A G, Zhang S W, Zou X N, Huxley R, Chen W Q, Williams G. The association between lung cancer incidence and ambient air pollution in China: a spatiotemporal analysis. Environmental Research, 2016, 144: 60-65 DOI:10.1016/j.envres.2015.11.004

4. Novellis P, Bottoni E, Voulaz E, Cariboni U, Testori A, Bertolaccini L, Giordano L, Dieci E, Granato L, Vanni E, Montorsi M, Alloisio M, Veronesi G. Robotic surgery, video-assisted thoracic surgery, and open surgery for early stage lung cancer: comparison of costs and outcomes at a single institute. Journal of Thoracic Disease, 2018, 10(2): 790-798 DOI:10.21037/jtd.2018.01.123

5. Sanchez-Lorente D, Guzman R, Boada M, Carriel N, Guirao A, Molins L. Is it appropriate to perform video-assisted thoracoscopic surgery for advanced lung cancer? Future Oncology, 2018, 14(6s): 29-31 DOI:10.2217/fon-2017-0388

6. Chen S, Geraci T C, Cerfolio R J. Techniques for lung surgery: a review of robotic lobectomy. Expert Review of Respiratory Medicine, 2018, 12(4): 315-322 DOI:10.1080/17476348.2018.1448270

7. Kent M, Wang T, Whyte R, Curran T, Flores R, Open Gangadharan S., video-assisted thoracic surgery, and lobectomy robotic: review of a national database. The Annals of Thoracic Surgery, 2014, 97(1): 236-244 DOI:10.1016/j.athoracsur.2013.07.117

8. Qin Z, Tai Y, Xia C, Peng J, Huang X, Chen Z, Li Q, Shi J. Towards virtual VATS, face, and construct evaluation for peg transfer training of box, VR, AR, and MR trainer. Journal of Healthcare Engineering, 2019, 6813719 DOI:10.1155/2019/6813719

9. Herron J. Augmented reality in medical education and training. Journal of Electronic Resources in Medical Libraries, 2016, 13(2): 51-55 DOI:10.1080/15424065.2016.1175987

10. Barsom E Z, Graafland M, Schijven M P. Systematic review on the effectiveness of augmented reality applications in medical training. Surgical Endoscopy, 2016, 30(10): 4174-4183 DOI:10.1007/s00464-016-4800-6

11. Loukas C. Surgical simulation training systems: box trainers, virtual reality and augmented reality simulators. International Journal of Advanced Robotics and Automation, 2016, 1(2): 1-9 DOI:10.15226/2473-3032/1/2/00109

12. Wu J, Westermann R, Dick C. A survey of physically based simulation of cuts in deformable bodies. Computer Graphics Forum, 2015, 34(6): 161-187 DOI:10.1111/cgf.12528

13. Maciel A, Liu Y Q, Ahn W, Singh T P, Dunnican W, De S. Development of the VBLaST: a virtual basic laparoscopic skill trainer. The International Journal of Medical Robotics + Computer Assisted Surgery: MRCAS, 2008, 4(2): 131-138 DOI:10.1002/rcs.185

14. Pan J J, Chang J, Yang X, Liang H, Zhang J J, Qureshi T, Howell R, Hickish T. Virtual reality training and assessment in laparoscopic rectum surgery. The International Journal of Medical Robotics + Computer Assisted Surgery, 2015, 11(2): 194-209 DOI:10.1002/rcs.1582

15. Kibsgaard M , Thomsen K K , Kraus M. Simulation of surgical cutting in deformable bodies using a game engine. In: Proceedings of the 9th International Conference on Computer Graphics Theory and Applications. Lisbon, Portugal, SCITEPRESS-Science and and Technology Publications, 2014 DOI:10.5220/0004670403420347

16. Paulus C J, Untereiner L, Courtecuisse H, Cotin S, Cazier D. Virtual cutting of deformable objects based on efficient topological operations. The Visual Computer, 2015, 31(6/7/8): 831-841 DOI:10.1007/s00371-015-1123-x

17. Zhu B, Gu L X. A hybrid deformable model for real-time surgical simulation. Computerized Medical Imaging and Graphics, 2012, 36(5): 356-365 DOI:10.1016/j.compmedimag.2012.03.001

18. Pons-Moll G, Romero J, Mahmood N, Black M J. Dyna. ACM Transactions on Graphics, 2015, 34(4): 1-14 DOI:10.1145/2766993

19. Lee J H, Kim H, Kim J H, Lee S H. Soft implantable microelectrodes for future medicine: prosthetics, neural signal recording and neuromodulation. Lab on a Chip, 2016, 16(6): 959-976 DOI:10.1039/c5lc00842e

20. Pan J J, Yan S Z, Qin H, Hao A M. Real-time dissection of organs via hybrid coupling of geometric metaballs and physics-centric mesh-free method. The Visual Computer, 2018, 34(1): 105-116 DOI:10.1007/s00371-016-1317-x

21. Berndt I, Torchelsen R, Maciel A. Efficient surgical cutting with position-based dynamics. IEEE Computer Graphics and Applications, 2017, 37(3): 24-31 DOI:10.1109/mcg.2017.45

22. Niebe S, Erleben K. Numerical methods for linear complementarity problems in physics-based animation. Synthesis Lectures on Computer Graphics and Animation, 2015, 7(1): 1-159 DOI:10.2200/s00621ed1v01y201412cgr018

23. Wang H M, O'Brien J, Ramamoorthi R. Multi-resolution isotropic strain limiting. ACM Transactions on Graphics, 2010, 29(6): 1-10 DOI:10.1145/1882261.1866182

24. Müller M, Chentanez N, Kim T Y, Macklin M. Strain based dynamics. Proc. ACM SIGGRAPH/Eurographics Symp. Comput. Animat., 2014, 149-157

25. Bender J, Koschier D, Charrier P, Weber D. Position-based simulation of continuous materials. Computers & Graphics, 2014, 44: 1-10 DOI:10.1016/j.cag.2014.07.004

26. Balasubramanian R, Santos V J. The human hand as an inspiration for robot hand development. Cham: Springer International Publishing, 2014 DOI:10.1007/978-3-319-03017-3

27. L'Orsa R, MacNab C J B, Tavakoli M. Introduction to haptics for neurosurgeons. Neurosurgery, 2013, 72: A139-A153 DOI:10.1227/neu.0b013e318273a1a3

28. Kim M, Kim J, Lee Y, Lee D. On the passivity of mechanical integrators in haptic rendering. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, IEEE, 2017, 446-452 DOI:10.1109/icra.2017.7989058

29. Ang Q Z, Horan B, Najdovski Z, Nahavandi S. Grasping virtual objects with multi-point haptics. 2011 IEEE Virtual Reality Conference, 2011, 189-190 DOI:10.1109/vr.2011.5759462

30. Jeon S, Harders M. Haptic tumor augmentation: exploring multi-point interaction. IEEE Transactions on Haptics, 2014, 7(4): 477-485 DOI:10.1109/toh.2014.2330300

31. Chen L, Day T W, Tang W, John N W. Recent developments and future challenges in medical mixed reality. In: 2017 IEEE International Symposium on Mixed and Augmented Reality(ISMAR). Nantes, France, IEEE, 2017, 123-135 DOI:10.1109/ismar.2017.29


1. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

2. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

3. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

4. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

5. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

6. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

7. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

8. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

9. Pengfei HAN, Gang ZHAO, A review of edge-based 3D tracking of rigid objects Virtual Reality & Intelligent Hardware 2019, 1(6): 580-596

10. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

11. Lingfei ZHU, Qi CAO, Yiyu CAI, Development of augmented reality serious games with a vibrotactile feedback jacket Virtual Reality & Intelligent Hardware 2020, 2(5): 454-470

12. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

13. Jie REN, Chun YU, Yueting WENG, Chengchi ZHOU, Yuanchun SHI, Design and evaluation of window management operations in AR headset+smartphone interface Virtual Reality & Intelligent Hardware 2022, 4(2): 115-131

14. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

15. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460

16. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

17. Rafik HAMZA, Minh-Son DAO, Privacy-preserving deep learning techniques for wearable sensor-based big data applications Virtual Reality & Intelligent Hardware 2022, 4(3): 210-222

18. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246

19. Ahmed L ALYOUSIFY, Ramadhan J MSTAFA, AR-assisted children book for smart teaching and learning of Turkish alphabets Virtual Reality & Intelligent Hardware 2022, 4(3): 263-277