Multi-source data-based 3D digital preservation of large-scale ancient chinese architecture: A case report

Abstract

The 3D digitalization and documentation of ancient Chinese architecture is challenging because of architectural complexity and structural delicacy. To generate complete and detailed models of this architecture, it is better to acquire, process, and fuse multi-source data instead of single-source data. In this paper, we describe our work on 3D digital preservation of ancient Chinese architecture based on multi-source data. We first briefly introduce two surveyed ancient Chinese temples, Foguang Temple and Nanchan Temple. Then, we report the data acquisition equipment we used and the multi-source data we acquired. Finally, we provide an overview of several applications we conducted based on the acquired data, including ground and aerial image fusion, image and LiDAR (light detection and ranging) data fusion, and architectural scene surface reconstruction and semantic modeling. We believe that it is necessary to involve multi-source data for the 3D digital preservation of ancient Chinese architecture, and that the work in this paper will serve as a heuristic guideline for the related research communities.

Keyword

Ancient Chinese architecture ; 3D digital preservation ; Multi-source data acquisition ; Architectural scene modeling

Cite this article

Xiang GAO, Hainan CUI, Lingjie ZHU, Tianxin SHI, Shuhan SHEN. Multi-source data-based 3D digital preservation of large-scale ancient chinese architecture: A case report. Virtual Reality & Intelligent Hardware, 2019, 1(5): 525-541 DOI:10.1016/j.vrih.2019.08.003

References

1. Cui H N, Gao X, Shen S H, Hu Z Y. HSfM: hybrid structure-from-motion. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.257

2. Shen S H. Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes. IEEE Transactions on Image Processing, 2013, 22(5): 1901–1914 DOI:10.1109/tip.2013.2237921

3. Ummenhofer B, Brox T. Global, dense multiscale reconstruction for a billion points. International Journal of Computer Vision, 2017, 125(1/2/3): 82–94 DOI:10.1007/s11263-017-1017-7

4. Zhu L J, Shen S H, Gao X, Hu Z Y. Large scale urban scene modeling from MVS meshes//Computer Vision―ECCV 2018. Springer International Publishing, 2018, 640–655 DOI:10.1007/978-3-030-01252-6_38

5. Choi S, Zhou Q Y, Koltun V. Robust reconstruction of indoor scenes. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015 DOI:10.1109/cvpr.2015.7299195

6. Dai A, Nießner M, Zollhöfer M, Izadi S, Theobalt C. BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics, 2016, 36(3): 24 DOI:10.1145/3054739

7. Dong W, Wang Q Y, Wang X, Zha H B. PSDF fusion: probabilistic signed distance function for on-the-fly 3D data fusion and scene reconstruction//Computer Vision―ECCV, 2018. Cham: Springer International Publishing, 2018, 714–730 DOI:10.1007/978-3-030-01240-3_43

8. Liu Y D, Gao W, Hu Z Y. Geometrically stable tracking for depth images based 3D reconstruction on mobile devices. ISPRS Journal of Photogrammetry and Remote Sensing 2018, 143, 222–232 DOI:10.1016/j.isprsjprs.2018.03.009

9. Zheng Q, Sharf A, Wan G, Li Y, Mitra N J, Cohen-Or D, Chen B. Non-local scan consolidation for 3D urban scenes. ACM Trans Graph, 2010, 29(4), 94–1 DOI:10.1145/1833349.1778831

10. Nan L, Sharf A, Zhang H, Cohen-Or D, Chen B. Smartboxes for interactive urban reconstruction. ACM Transactions on Graphics, 2010, 29(4): 93 DOI:10.1145/1833349.1778830

11. Vanegas C A, Aliaga D G, Benes B. Automatic extraction of Manhattan-world building masses from 3D laser range scans. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(10): 1627–1637 DOI:10.1109/tvcg.2012.30

12. Li M L, Wonka P, Nan L L. Manhattan-world urban reconstruction from point clouds// Computer Vision―ECCV 2016. Cham: Springer International Publishing, 2016, 54–69 DOI:10.1007/978-3-319-46493-0_4

13. Gao X, Shen S H, Zhou Y, Cui H N, Zhu L J, Hu Z Y. Ancient Chinese architecture 3D preservation by merging ground and aerial point clouds. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 143, 72–84 DOI:10.1016/j.isprsjprs.2018.04.023

14. Gao X, Shen S, Zhu L, Shi T, Wang Z, Hu Z. Complete scene reconstruction by merging images and laser scans. arXiv preprint, arXiv:1904.09568

15. Zhou Y, Shen S H, Hu Z Y. Detail preserved surface reconstruction from point cloud. Sensors, 2019, 19(6): 1278 DOI:10.3390/s19061278

16. Zhou Y, Shen S H, Hu Z Y. Fine-level semantic labeling of large-scale 3D model by active learning. In: 2018 International Conference on 3D Vision (3DV). New York, USA, IEEE, 2018 DOI:10.1109/3dv.2018.00066

17. Besl P J, McKay N D. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 239–256 DOI:10.1109/34.121791

18. Shan Q, Wu C C, Curless B, Furukawa Y, Hernandez C, Seitz S M. Accurate geo-registration by ground-to-aerial image matching. In: 2014 2nd International Conference on 3D Vision. New York, USA, IEEE, 2014 DOI:10.1109/3dv.2014.69

19. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 DOI:10.1023/b:visi.0000029664.99615.94

20. Mikolajczyk K, Schmid C. A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(10): 1615–1630 DOI:10.1109/tpami.2005.188

21. Umeyama S. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1991, 13(4): 376–380 DOI:10.1109/34.88573

22. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

23. Zhou Y, Shen S H, Gao X, Hu Z Y. Accurate mesh-based alignment for ground and aerial multi-view stereo models. In: 2017 IEEE International Conference on Image Processing (ICIP). New York, USA, IEEE, 2017 DOI:10.1109/icip.2017.8296758

24. Gao X, Hu L H, Cui H N, Shen S H, Hu Z Y. Accurate and efficient ground-to-aerial model alignment. Pattern Recognition, 2018, 76, 288–302 DOI:10.1016/j.patcog.2017.11.003

25. Agarwal S, Snavely N, Seitz S M, Szeliski R. Bundle adjustment in the large. In: Proceedings of the 11th European conference on Computer vision: Part II, 2010, 29–42

26. Bastonero P, Donadio E, Chiabrando F, Spanò A. Fusion of 3D models derived from TLS and image-based techniques for CH enhanced documentation. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014, II-5, 73–80 DOI:10.5194/isprsannals-ii-5-73-2014

27. Russo M, Manferdini A M. Integration of image and range-based techniques for surveying complex architectures. ISPRS Annals of Photogrammetry. Remote Sensing and Spatial Information Sciences, 2014, II-5, 305–312 DOI:10.5194/isprsannals-ii-5-305-2014

28. Altuntas C. Integration of point clouds originated from laser scaner and photogrammetric images for visualization of complex details of historical buildings. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2015, XL-5/W4, 431–435 DOI:10.5194/isprsarchives-xl-5-w4-431-2015

29. Sinha S N, Mordohai P, Pollefeys M. Multi-view stereo via graph cuts on the dual of an adaptive tetrahedral mesh. In: 2007 IEEE 11th International Conference on Computer Vision. Rio de Janeiro, Brazil, IEEE, 2007 DOI:10.1109/iccv.2007.4408997

30. Jancosek M, Pajdla T. Exploiting visibility information in surface reconstruction to preserve weakly supported surfaces. International Scholarly Research Notices, 2014, 2014, 1–20 DOI:10.1155/2014/798595

31. Hane C, Zach C, Cohen A, Pollefeys M. Dense semantic 3D reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(9): 1730–1743 DOI:10.1109/tpami.2016.2613051

32. Vu H H, Labatut P, Pons J P, Keriven R. High accuracy and visibility-consistent dense multiview stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 889–901 DOI:10.1109/tpami.2011.172

33. Blaha M, Vogel C, Richard A, Wegner J D, Pock T, Schindler K. Large-scale semantic 3D reconstruction: an adaptive multi-resolution model for multi-class volumetric labeling. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016 DOI:10.1109/cvpr.2016.346

34. Cherabier I, Hane C, Oswald M R, Pollefeys M. Multi-label semantic 3D reconstruction using voxel blocks. In: 2016 Fourth International Conference on 3D Vision (3DV). Stanford, CA, USA, IEEE, 2016 DOI:10.1109/3dv.2016.68

35. Valentin J P C, Sengupta S, Warrell J, Shahrokni A, Torr P H S. Mesh based semantic modelling for indoor and outdoor scenes. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, IEEE, 2013 DOI:10.1109/cvpr.2013.269

36. Rouhani M, Lafarge F, Alliez P. Semantic segmentation of 3D textured meshes for urban scene analysis. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 123, 124–139 DOI:10.1016/j.isprsjprs.2016.12.001

37. McCormac J, Handa A, Davison A, Leutenegger S. SemanticFusion: Dense 3D semantic mapping with convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore, IEEE, 2017 DOI:10.1109/icra.2017.7989538

38. Schonberger J L, Frahm J M. Structure-from-motion revisited. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016 DOI:10.1109/cvpr.2016.445

39. Charles R Q, Hao S, Mo K C, Guibas L J. PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, IEEE, 2017 DOI:10.1109/cvpr.2017.16

40. Qi C R, Yi L, Su H, Guibas L J. PointNet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems (NIPS), 2017, 5099–5108