Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2022, 4(1): 55-75

Published Date:2022-2-20 DOI: 10.1016/j.vrih.2022.01.004

Virtual-reality-based digital twin of office spaces with social distance measurement feature

Abstract

Background
Social distancing is an effective way to reduce the spread of the SARS-CoV-2 virus. Many students and researchers have already attempted to use computer vision technology to automatically detect human beings in the field of view of a camera and help enforce social distancing. However, because of the present lockdown measures in several countries, the validation of computer vision systems using large-scale datasets is a challenge.
Methods
In this paper, a new method is proposed for generating customized datasets and validating deep-learning-based computer vision models using virtual reality (VR) technology. Using VR, we modeled a digital twin (DT) of an existing office space and used it to create a dataset of individuals in different postures, dresses, and locations. To test the proposed solution, we implemented a convolutional neural network (CNN) model for detecting people in a limited-sized dataset of real humans and a simulated dataset of humanoid figures.
Results
We detected the number of persons in both the real and synthetic datasets with more than 90% accuracy, and the actual and measured distances were significantly correlated (r=0.99). Finally, we used intermittent-layer- and heatmap-based data visualization techniques to explain the failure modes of a CNN.
Conclusions
A new application of DTs is proposed to enhance workplace safety by measuring the social distance between individuals. The use of our proposed pipeline along with a DT of the shared space for visualizing both environmental and human behavior aspects preserves the privacy of individuals and improves the latency of such monitoring systems because only the extracted information is streamed.

Keyword

Virtual environment ; Digital twin ; 3D visualization ; Convolutional neural network ; Object detection ; Social distancing

Cite this article

Abhishek MUKHOPADHYAY, G S Rajshekar REDDY, KamalPreet Singh SALUJA, Subhankar GHOSH, Anasol PEÑA-RIOS, Gokul GOPAL, Pradipta BISWAS. Virtual-reality-based digital twin of office spaces with social distance measurement feature. Virtual Reality & Intelligent Hardware, 2022, 4(1): 55-75 DOI:10.1016/j.vrih.2022.01.004

References

1. Glaessgen E, Stargel D. The digital twin paradigm for future NASA and US air force vehicles. In: 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference20th AIAA/ASME/AHS Adaptive Structures Conference14th AIAA. Honolulu, Hawaii, Reston, Virigina, 2012,1818 DOI:10.2514/6.2012-1818

2. Tao F, Zhang H, Liu A, Nee A Y C. Digital twin in industry: state-of-the-art. IEEE Transactions on Industrial Informatics, 2019, 15(4): 2405–2415 DOI:10.1109/tii.2018.2873186

3. Khajavi S H, Motlagh N H, Jaribion A, Werner L C, Holmström J. Digital twin: vision, benefits, boundaries, and creation for buildings. IEEE Access, 2019, 7: 147406–147419 DOI:10.1109/access.2019.2946515

4. Khelifi, I 2020, Azure Digital Twins: Powering the next generation of IoT connected solutions, Microsoft Azure Blog. [2020-8-5] Available from: https://azure.microsoft.com/en-in/blog/azure-digital-twins-powering-the-next-generation-of-iot-connected-solutions/

5. Steelcase 2019, Microsoft Corporation. [2020-8-5] Available from: https://customers.microsoft.com/en-US/story/steelcase-manufacturing-azureiot-en

6. ICONICS 2018, Microsoft Corporation. [2020-8-5] Available from: https://customers.microsoft.com/en-us/story/iconics-partner-professional-services-azure-iot

7. Milne G J, Xie S. The effectiveness of social distancing in mitigating COVID-19 spread: a modelling analysis. 2020

8. Vuorinen V, Aarnio M, Alava M, Alopaeus V, Atanasova N, Auvinen M, Balasubramanian N, Bordbar H, Erästö P, Grande R, Hayward N, Hellsten A, Hostikka S, Hokkanen J, Kaario O, Karvinen A, Kivistö I, Korhonen M, Österberg M. Modelling aerosol transport and virus exposure with numerical simulations in relation to SARS-CoV-2 transmission by inhalation indoors. Safety Science, 2020, 130: 104866 DOI:10.1016/j.ssci.2020.104866

9. Sharma, S 2020, Social distancing in the workplace: the new norm, Buro Happold. [2020-8-7] Available from: https://www.burohappold.com/articles/social-distancing-in-the-workplace/#

10. Fort J, Crespi A, Elion C, Kermanizadeh R, Wani P, Lange D. “Simulation + Coronavirus”, Unity Technologies White Paper 2020.

11. Google Sodar n.d., GoogleInc. [2020-8-7] Available from: https://sodar.withgoogle.com

12. BradPorter, Amazon introduces 'Distance Assistant', The Amazon Blog. [2020-8-9] Available from: https://blog.aboutamazon.com/operations/amazon-introduces-distance-assistant

13. Papageorgiou C, Poggio T. A trainable system for object detection. International Journal of Computer Vision, 2000, 38(1): 15–33 DOI:10.1023/a:1008162616689

14. Mohan A, Papageorgiou C, Poggio T. Example-based object detection in images by components. IEEE transactions on pattern analysis and machine intelligence, 2001, 23(4): 349-361 DOI:10.1109/34.917571

15. Viola P, Jones M J, Snow D. Detecting pedestrians using patterns of motion and appearance. International Journal of Computer Vision, 2005, 63(2): 153-161 DOI:10.1007/s11263-005-6644-8

16. Wu B, Nevatia R. Detection of multiple, partially occluded humans in a single image by Bayesian combination of edgelet part detectors. In: Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1. Beijing, China, IEEE, 2005, 90–97 DOI:10.1109/iccv.2005.74

17. Wang X Y, Han T X, Yan S C. An HOG-LBP human detector with partial occlusion handling. In: 2009 IEEE 12th International Conference on Computer Vision. Kyoto, Japan, IEEE, 2009, 32–39 DOI:10.1109/iccv.2009.5459207

18. Dalal N, Triggs B. Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05.SanDiego, CA, USA, IEEE, 2005, 886–893 DOI:10.1109/cvpr.2005.177

19. Girshick R, Felzenszwalb P, McAllester D. Object detection with grammar models. Advances in Neural Information Processing Systems, 2011, 24: 442-450

20. Zhu Q, Yeh M C, Cheng K T, Avidan S. Fast human detection using a cascade of histograms of oriented gradients. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. NewYork, NY, USA, IEEE, 2006, 1491–1498 DOI:10.1109/cvpr.2006.119

21. Felzenszwalb P F, Girshick R B, McAllester D, Ramanan D. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 2009, 32(9): 1627–1645 DOI:10.1109/tpami.2009.153

22. Sadeghi M A, Forsyth D. 30hz object detection with dpm v5. European Conference on Computer Vision. Springer, Cham, 2014, 65–79

23. Hosang J, Omran M, Benenson R, Schiele B. Taking a deeper look at pedestrians. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015, 4073–4082 DOI:10.1109/cvpr.2015.7299034

24. Zhang L, Lin L, Liang X, He K. Is faster R-CNN doing well for pedestrian detection? European conference on computer vision. Springer, Cham, 2016, 443–457

25. Cao J L, Pang Y W, Li X L. Learning multilayer channel features for pedestrian detection. IEEE Transactions on Image Processing, 2017, 26(7): 3210–3220 DOI:10.1109/tip.2017.2694224

26. Tian Y L, Luo P, Wang X G, Tang X O. Pedestrian detection aided by deep learning semantic tasks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015, 5079–5087 DOI:10.1109/cvpr.2015.7299143

27. Xu D, Ouyang W, Ricci E, et al. Learning cross-modal deep representations for robust pedestrian detection. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, 5363–5371

28. Wang X L, Xiao T T, Jiang Y N, Shao S, Sun J, Shen C H. Repulsion loss: detecting pedestrians in a crowd. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 7774–7783 DOI:10.1109/cvpr.2018.00811

29. Hagras H. Toward human-understandable, explainable AI. Computer, 2018, 51(9): 28–36 DOI:10.1109/mc.2018.3620965

30. Mukhopadhyay A, Mukherjee I, Biswas P. Decoding CNN based object classifier using visualization. In: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. Virtual Event DC USA, New York, NY, USA, ACM, 2020, 50–53 DOI:10.1145/3409251.3411721

31. Nikolakis N, Alexopoulos K, Xanthakis E, Chryssolouris G. The digital twin implementation for linking the virtual representation of human-based production tasks to their physical counterpart in the factory-floor. International Journal of Computer Integrated Manufacturing, 2019, 32(1): 1–12 DOI:10.1080/0951192x.2018.1529430

32. Ray tracing n.d., UnityTechnologies. [2020-5-10] Available from: https://unity.com/ray-tracing

33. Path tracing n.d., UnityTechnologies. [2020-5-10] Available from: https://docs.unity3d.com/Packages/com.unity.render-pipelines.high-definition@7.1/manual/Ray-Tracing-Path-Tracing.html

34. Murthy L R D. Multimodal interaction for real and virtual environments. In: International Conference on Intelligent User Interfaces, Proceedings IUI. Association for Computing Machinery, 2020, 29–30 DOI:10.1145/3379336.3381506

35. Mukhopadhyay A, Mukherjee I, Biswas P. Comparing CNNs for non-conventional traffic participants. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings. Utrecht Netherlands, New York, NY, USA, ACM, 2019, 171–175 DOI:10.1145/3349263.3351336

36. Punn N S, Sonbhadra S K, Agarwal S. Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques. 2020

37. Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016, 779–788 DOI:10.1109/cvpr.2016.91

38. Zeiler M D, Fergus R. Visualizing and understanding convolutional networks. European conference on computer vision. Springer, Cham, 2014, 818–833

39. Selvaraju R R, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy, IEEE, 2017: 618–626 DOI:10.1109/iccv.2017.74

40. Volk R, Stengel J, Schultmann F. Building Information Modeling (BIM) for existing buildings—Literature review and future needs. Automation in Construction, 2014, 38, 109–127 DOI:10.1016/j.autcon.2013.10.023

41. Kupriyanovsky V, Klimov A, Voropaev Y, Pokusaev O, Dobrynin A, Ponkin I, Lysogorsky A. Digital twins based on the development of BIM technologies, related ontologies, 5G, IoT, and mixed reality for use in infrastructure projects and IFRABIM. International Journal of Open Information Technologies, 2020, 8(3): 55–74

42. Lu Q C, Xie X, Heaton J, Parlikad A K, Schooling J. From BIM towards digital twin: strategy and future development for smart asset management. In: Service Oriented, Holonic and Multi-agent Manufacturing Systems for Industry of the Future. Cham: Springer International Publishing, 2019: 392–404 DOI:10.1007/978-3-030-27477-1_30

43. Tridify 2020, Tridify Ltd. [2020-8-6] Available from: https://www.tridify.com

44. PiXYZ Plugin n.d., PiXYZSoftware. [2020-8-6] Available from: https://www.pixyz-software.com/plugin/

45. Unity Reflect n.d., UnityTechnologies. [2020-8-6] Available from: https://unity.com/products/unity-reflect

46. Tao F, Zhang H, Liu A, Nee A Y. Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 2018, 15(4): 2405–2415

47. ProBuildern.d., UnityTechnologies. [2020-8-6] Available from: https://unity3d.com/unity/features/worldbuilding/probuilder

48. ProGridsn.d., UnityTechnologies. [2020-8-6] Available from: https://docs.unity3d.com/Packages/com.unity.progrids@3.0/manual/index.html

49. TurboSquid 2020. [2020-8-6] Available from: https://www.turbosquid.com/

50. Sketchfab 2020. [2020-8-6] Available from: https://sketchfab.com/

51. LeBlanc H. What are PBR Materials, E-on Software Blog. [2020-8-6] Available from: https://info.e-onsoftware.com/learning_vue/what-are-pbr-materials

52. NavMesh Agents n.d., UnityTechnologies. [2020-8-6] Available from: https://docs.unity3d.com/Manual/class-NavMeshAgent.html

53. Biswas P, Saluja K S, Arjun S, Murthy L, Prabhakar G, Sharma V K, Dv J S. COVID-19 data visualization through automatic phase detection. Digital Government: Research and Practice, 2020, 1(4): 25 DOI:10.1145/3411756

54. Woldstad J C. Digital human models for ergonomics. 2000

55. Mixamo. Adobe. [2020-8-27] Available from: https://www.mixamo.com/#/?page=1&type=Motion%2CMotionPack

56. Li X, Wang K F, Tian Y L, Yan L, Deng F, Wang F Y. The ParallelEye dataset: a large collection of virtual images for traffic vision research. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(6): 2072–2084 DOI:10.1109/tits.2018.2857566

57. Martinez-Gonzalez P, Oprea S, Garcia-Garcia A, Jover-Alvarez A, Orts-Escolano S, Garcia-Rodriguez J. UnrealROX: an extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation. Virtual Reality, 2020, 24(2): 271–288 DOI:10.1007/s10055-019-00399-5

58. Paisley J, Blei D, Jordan M. Variational Bayesian inference with stochastic search. arXiv preprint arXiv:1206.6430, 2012

59. Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015

60. Chen R T Q, Li X, Grosse R, Duvenaud D. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942, 2018

61. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. Advances in neural information processing systems, 2014, 27, 2672–2680

62. Shaham T R, Dekel T, Michaeli T. SinGAN: learning a generative model from a single natural image. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South), IEEE, 2019, 4569–4579 DOI:10.1109/iccv.2019.00467

63. Xian W Q, Sangkloy P, Agrawal V, Raj A, Lu J W, Fang C, Yu F, Hays J. TextureGAN: controlling deep image synthesis with texture patches. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 8456–8465 DOI:10.1109/cvpr.2018.00882

64. Zhang K, Zuo W M, Chen Y J, Meng D Y, Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 2017, 26(7): 3142–3155 DOI:10.1109/tip.2017.2662206

65. Zontak M, Irani M. Internal statistics of a single natural image. CVPR 2011, 2011, 977–984 DOI:10.1109/cvpr.2011.5995401

66. Zontak M, Mosseri I, Irani M. Separating signal from noise using patch recurrence across scales. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, IEEE, 2013, 1195–1202 DOI:10.1109/cvpr.2013.158

67. Denton E, Chintala S, Szlam A, Fergus R. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, 2015, 1486–1494

68. Huang X, Li Y X, Poursaeed O, Hopcroft J, Belongie S. Stacked generative adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017, 1866–1875 DOI:10.1109/cvpr.2017.202

69. Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017

70. Barnett S A. Convergence problems with generative adversarial networks (gans). arXiv preprint arXiv:1806.11382, 2018

71. Park S W, Huh J H, Kim J C. BEGAN v3: avoiding mode collapse in GANs using variational inference. Electronics, 2020, 9(4): 688 DOI:10.3390/electronics9040688

72. Kingma D P, Welling M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

73. Open Images Dataset V4. [2020-8-27] Available from: https://github.com/openimages/dataset

74. Bertoni L, Kreiss S, Alahi A. Perceiving humans: from monocular 3D localization to social distancing. IEEE Transactions on Intelligent Transportation Systems, 2021, (99): 1–18 DOI:10.1109/tits.2021.3069376

Related

1. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

2. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

3. Rachel HUANG, Carisa HARRIS-ADAMSON, Dan ODELL, David REMPEL, Design of finger gestures for locomotion in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 1-9

4. Jun-Hao YIN, Chin-Boon CHNG, Pooi-Mun WONG, Nicholas HO, Matthew CHUA, Chee-Kong CHUI, VR and AR in human performance researchAn NUS experience Virtual Reality & Intelligent Hardware 2020, 2(5): 381-393

5. Zixiang ZHAO, Quanwei ZHOU, Xiaoguang HAN, Lili WANG, Dynamic targets searching assistance based on virtual camera priority Virtual Reality & Intelligent Hardware 2021, 3(6): 484-500

6. Carlos ARCE-LOPERA, María José ARIAS, Gustavo CORRALES, Training birdsong recognition using virtual reality Virtual Reality & Intelligent Hardware 2021, 3(5): 397-406

7. Vivian GÓMEZ, Kelly PEÑARANDA, Pablo FIGUEROA, Lessons learned from requirements gathering for virtual reality simulators Virtual Reality & Intelligent Hardware 2021, 3(5): 407-422