Flow-based SLAM: From geometry computation to learning

Abstract

Simultaneous localization and mapping (SLAM) has attracted considerable research interest from the robotics and computer-vision communities for >30 years. With steady and progressive efforts being made, modern SLAM systems allow robust and online applications in real-world scenes. We examined the evolution of this powerful perception tool in detail and noticed that the insights concerning incremental computation and temporal guidance are persistently retained. Herein, we denote this temporal continuity as a flow basis and present for the first time a survey that specifically focuses on the flow-based nature, ranging from geometric computation to the emerging learning techniques. We start by reviewing two essential stages for geometric computation, presenting the de facto standard pipeline and problem formulation, along with the utilization of temporal cues. The recently emerging techniques are then summarized, covering a wide range of areas, such as learning techniques, sensor fusion, and continuous-time trajectory modeling. This survey aims at arousing public attention on how robust SLAM systems benefit from a continuously observing nature, as well as the topics worthy of further investigation for better utilizing the temporal cues.

Keyword

Simultaneous localization and mapping ; Visual odometry ; Deep learning ; Flow basis ; Sensor fusion ; Augmented reality

Cite this article

Zike YAN, Hongbin ZHA. Flow-based SLAM: From geometry computation to learning. Virtual Reality & Intelligent Hardware, 2019, 1(5): 435-460 DOI:10.1016/j.vrih.2019.09.001

References

1. Schonberger J L, Radenovic F, Chum O, Frahm J M. From single image query to detailed 3D reconstruction. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015 DOI:10.1109/cvpr.2015.7299148

2. Engel J, Schöps T, Cremers D. LSD-SLAM: large-scale direct monocular SLAM//Computer Vision―ECCV 2014. Cham: Springer International Publishing, 2014, 834–849 DOI:10.1007/978-3-319-10605-2_54

3. Newcombe R A, Lovegrove S J, Davison A J. DTAM: Dense tracking and mapping in real-time. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011 DOI:10.1109/iccv.2011.6126513

4. Saputra M R U, Markham A, Trigoni N. Visual SLAM and structure from motion in dynamic environments. ACM Computing Surveys, 2018, 51(2): 1–36 DOI:10.1145/3177853

5. Strasdat H, Montiel J M M, Davison A J. Real-time monocular SLAM: Why filter? In: 2010 IEEE International Conference on Robotics and Automation. Anchorage, AK, NewYork, USA, IEEE, 2010 DOI:10.1109/robot.2010.5509636

6. Bailey T, Durrant-Whyte H. Simultaneous localization and mapping (SLAM): Part II. IEEE Robotics & Automation Magazine, 2006, 13(3): 108–117 DOI:10.1109/mra.2006.1678144

7. Durrant-Whyte H, Bailey T. Simultaneous localization and mapping: Part I. IEEE Robotics & Automation Magazine, 2006, 13(2): 99–110 DOI:10.1109/mra.2006.1638022

8. Cadena C, Carlone L, Carrillo H, Latif Y, Scaramuzza D, Neira J, Reid I, Leonard J J. Past, present, and future of simultaneous localization and mapping: toward the robust-perception age. IEEE Transactions on Robotics, 2016, 32(6): 1309–1332 DOI:10.1109/tro.2016.2624754

9. Dellaert F, Kaess M. Factor graphs for robot perception. Foundations and Trends in Robotics, 2017, 6(1/2): 1–139 DOI:10.1561/2300000043

10. Grisetti G, Kummerle R, Stachniss C, Burgard W. A tutorial on graph-based SLAM. IEEE Intelligent Transportation Systems Magazine, 2010, 2(4): 31–43 DOI:10.1109/mits.2010.939925

11. Bresson G, Alsayed Z, Yu L, Glaser S. Simultaneous localization and mapping: A survey of current trends in autonomous driving. IEEE Transactions on Intelligent Vehicles, 2017, 2(3): 194–220 DOI:10.1109/tiv.2017.2749181

12. Haarbach A, Birdal T, Ilic S. Survey of higher order rigid body motion interpolation methods for keyframe animation and continuous-time trajectory estimation. In: 2018 International Conference on 3D Vision (3DV). Verona, NewYork, USA, IEEE, 2018 DOI:10.1109/3dv.2018.00051

13. Li J, Yang B, Chen D, Wang N, Zhang G F, Bao H J. Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality. Virtual Reality and Intelligent Hardware, 2019, 1(1): 386–410 DOI:10.1016/j.vrih.2019.07.002

14. Mur-Artal R, Montiel J M M, Tardos J D. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 2015, 31(5): 1147–1163 DOI:10.1109/tro.2015.2463671

15. Zhou H Z, Zou D P, Pei L, Ying R D, Liu P L, Yu W X. StructSLAM: visual SLAM with building structure lines. IEEE Transactions on Vehicular Technology2015, 64(4): 1364–1375 DOI:10.1109/tvt.2015.2388780

16. Engel J, Stuckler J, Cremers D. Large-scale direct SLAM with stereo cameras. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany, IEEE, 2015 DOI:10.1109/iros.2015.7353631

17. Whelan T, Salas-Moreno R F, Glocker B, Davison A J, Leutenegger S. ElasticFusion: Real-time dense SLAM and light source estimation. The International Journal of Robotics Research, 2016, 35(14): 1697–1716 DOI:10.1177/0278364916669237

18. Davison A J, Reid I D, Molton N D, Stasse O. MonoSLAM: real-time single camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 1052–1067 DOI:10.1109/tpami.2007.1049

19. Klein G, Murray D. Parallel tracking and mapping for small AR workspaces. In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan, IEEE, 2007 DOI:10.1109/ismar.2007.4538852

20. Pumarola A, Vakhitov A, Agudo A, Sanfeliu A, Moreno-Noguer F. PL-SLAM: Real-time monocular visual SLAM with points and lines. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore, IEEE, 2017 DOI:10.1109/icra.2017.7989522

21. von Gioi R G, Jakubowicz J, Morel J M, Randall G. LSD: A fast line segment detector with a false detection control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(4): 722–732 DOI:10.1109/tpami.2008.300

22. Zhang L L, Koch R. An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency. Journal of Visual Communication and Image Representation, 2013, 24(7): 794–805 DOI:10.1016/j.jvcir.2013.05.006

23. Qin T, Li P L, Shen S J. VINS-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018, 34(4): 1004–1020 DOI:10.1109/tro.2018.2853729

24. Lucas B D, Kanade T. An iterative image registration technique with an application to stereo vision. Proceedings of the 7th international joint conference on Artificial intelligence,1981, 2, 674–679

25. Engel J, Koltun V, Cremers D. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3): 611–625 DOI:10.1109/tpami.2017.2658577

26. Mur-Artal R, Tardos J D. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 2017, 33(5): 1255–1262 DOI:10.1109/tro.2017.2705103

27. Galvez-López D, Tardos J D. Bags of binary words for fast place recognition in image sequences. IEEE Transactions on Robotics, 2012, 28(5): 1188–1197 DOI:10.1109/tro.2012.2197158

28. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

29. Baker S, Matthews I. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision, 2004, 56(3): 221–255 DOI:10.1023/b:visi.0000011205.11775.fd

30. Peasley B, Birchfield S. Replacing projective data association with Lucas-kanade for KinectFusion. In: 2013 IEEE International Conference on Robotics and Automation. Karlsruhe, Germany, IEEE, 2013 DOI:10.1109/icra.2013.6630640

31. Vidal A R, Rebecq H, Horstschaefer T, Scaramuzza D. Ultimate SLAM? combining events, images, and IMU for robust visual SLAM in HDR and high-speed scenarios. IEEE Robotics and Automation Letters, 2018, 3(2): 994–1001 DOI:10.1109/lra.2018.2793357

32. Forster C, Pizzoli M, Scaramuzza D. SVO: Fast semi-direct monocular visual odometry. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). HongKong, China, IEEE, 2014 DOI:10.1109/icra.2014.6906584

33. Kerl C, Sturm J, Cremers D. Robust odometry estimation for RGB-D cameras. In: 2013 IEEE International Conference on Robotics and Automation. Karlsruhe, Germany, 2013 DOI:10.1109/icra.2013.6631104

34. Park J, Zhou Q Y, Koltun V. Colored point cloud registration revisited. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, NewYork, USA, IEEE, 2017 DOI:10.1109/iccv.2017.25

35. Zhou Q Y, Park J, Koltun V. Fast global registration//Computer Vision―ECCV 2016. Cham: Springer International Publishing, 2016, 766–782 DOI:10.1007/978-3-319-46475-6_47

36. Glocker B, Shotton J, Criminisi A, Izadi S. Real-time RGB-D camera relocalization via randomized ferns for keyframe encoding. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(5): 571–583 DOI:10.1109/tvcg.2014.2360403

37. Cummins M, Newman P. FAB-MAP: probabilistic localization and mapping in the space of appearance. The International Journal of Robotics Research, 2008, 27(6): 647–665 DOI:10.1177/0278364908090961

38. Klein G, Murray D. Improving the agility of keyframe-based SLAM//Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, 802–815 DOI:10.1007/978-3-540-88688-4_59

39. Lowry S, Sunderhauf N, Newman P, Leonard J J, Cox D, Corke P, Milford M J. Visual place recognition: A survey. IEEE Transactions on Robotics, 2016, 32(1): 1–19 DOI:10.1109/tro.2015.2496823

40. Maddern W, Milford M, Wyeth G. CAT-SLAM: probabilistic localisation and mapping using a continuous appearance-based trajectory. The International Journal of Robotics Research, 2012, 31(4): 429–451 DOI:10.1177/0278364912438273

41. Milford M J, Wyeth G F. SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights. In: 2012 IEEE International Conference on Robotics and Automation. StPaul, MN, USA, IEEE, 2012 DOI:10.1109/icra.2012.6224623

42. Angeli A, Filliat D, Doncieux S, Meyer J A. Fast and incremental method for loop-closure detection using bags of visual words. IEEE Transactions on Robotics, 2008, 24(5): 1027–1037 DOI:10.1109/tro.2008.2004514

43. Kawewong A, Tongprasit N, Tangruamsub S, Hasegawa O. Online and incremental appearance-based SLAM in highly dynamic environments. The International Journal of Robotics Research, 2011, 30(1): 33–55 DOI:10.1177/0278364910371855

44. Khan S, Wollherr D. IBuILD: Incremental bag of Binary words for appearance based loop closure detection. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). Seattle, WA, USA, IEEE, 2015 DOI:10.1109/icra.2015.7139959

45. Tsintotas K A, Bampis L, Gasteratos A. Assigning visual words to places for loop closure detection. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane, QLD, IEEE, 2018 DOI:10.1109/icra.2018.8461146

46. Davison A J. Real-time simultaneous localisation and mapping with a single camera. In: Proceedings Ninth IEEE International Conference on Computer Vision. Nice, France, IEEE, 2003 DOI:10.1109/iccv.2003.1238654

47. Holmes S, Klein G, Murray D W. A square root unscented kalman filter for visual monoSLAM. In: 2008 IEEE International Conference on Robotics and Automation. Pasadena, CA, USA, IEEE, 2008 DOI:10.1109/robot.2008.4543780

48. Du J J, Carlone L, Kaouk Ng M, Bona B, Indri M. A comparative study on active SLAM and autonomous exploration with Particle Filters. In: 2011 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM). Budapest, Hungary, IEEE, 2011 DOI:10.1109/aim.2011.6027142

49. Sibley G, Matthies L, Sukhatme G. Sliding window filter with application to planetary landing. Journal of Field Robotics, 2010, 27(5): 587–608 DOI:10.1002/rob.20360

50. Bailey T, Nieto J, Guivant J, Stevens M, Nebot E. Consistency of the EKF-SLAM algorithm. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. Beijing, China, IEEE, 2006 DOI:10.1109/iros.2006.281644

51. Dissanayake G, Huang S D, Wang Z, Ranasinghe R. A review of recent developments in Simultaneous Localization and Mapping. In: 2011 6th International Conference on Industrial and Information Systems. Kandy, SriLanka, IEEE, 2011 DOI:10.1109/iciinfs.2011.6038117

52. Huang S D, Dissanayake G. A critique of current developments in simultaneous localization and mapping. International Journal of Advanced Robotic Systems, 2016, 13(5): 172988141666948 DOI:10.1177/1729881416669482

53. Huang G P, Mourikis A I, Roumeliotis S I. A first-estimates Jacobian EKF for improving SLAM consistency// Experimental Robotics. Berlin, Heidelberg, 2009, 373–382 DOI:10.1007/978-3-642-00196-3_43

54. Montemerlo M, Thrun S, Koller D, Wegbreit B. FastSLAM: A factored solution to the simultaneous localization and mapping problem. In: National Conf. on Artificial Intelligence (AAAI), 2002, 593598

55. Thrun S, Liu Y F, Koller D, Ng A Y, Ghahramani Z, Durrant-Whyte H. Simultaneous localization and mapping with sparse extended information filters. The International Journal of Robotics Research, 2004, 23(7/8): 693–716 DOI:10.1177/0278364904045479

56. Huang S D, Wang Z, Dissanayake G. Sparse local submap joining filter for building large-scale maps. IEEE Transactions on Robotics, 2008, 24(5): 1121–1130 DOI:10.1109/tro.2008.2003259

57. Lenac K, Ćesić J, Marković I, Petrović I. Exactly sparse delayed state filter on Lie groups for long-term pose graph SLAM. The International Journal of Robotics Research, 2018, 37(6): 585–610 DOI:10.1177/0278364918767756

58. Thrun S, Burgard W, Fox D. Probabilistic robotics. MIT press, 2005

59. Dellaert F, Kaess M. Square root SAM: simultaneous localization and mapping via square root information smoothing. The International Journal of Robotics Research, 2006, 25(12): 1181–1203 DOI:10.1177/0278364906072768

60. Kummerle R, Grisetti G, Strasdat H, Konolige K, Burgard W. G2o: A general framework for graph optimization. In: 2011 IEEE International Conference on Robotics and Automation. Shanghai, China, IEEE, 2011 DOI:10.1109/icra.2011.5979949

61. Agarwal S, K.Others Mierle, CeresSolver. 2015

62. Sibley G, Sukhatme G S, Matthies L. Constant time sliding window filter SLAM as a basis for metric visual perception. In: IEEE International Conference on Robotics and Automation Workshop. 2007

63. Dong-Si T C, Mourikis A I. Motion tracking with fixed-lag smoothing: Algorithm and consistency analysis. In: 2011 IEEE International Conference on Robotics and Automation2. Shanghai, China, IEEE, 2011 DOI:10.1109/icra.2011.5980267

64. Huang G P, Mourikis A I, Roumeliotis S I. An observability-constrained sliding window filter for SLAM. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. SanFrancisco, CA, USA, IEEE, 2011 DOI:10.1109/iros.2011.6095161

65. Kaess M, Ranganathan A, Dellaert F. ISAM: incremental smoothing and mapping. IEEE Transactions on Robotics, 2008, 24(6): 1365–1378 DOI:10.1109/tro.2008.2006706

66. Wang X P, Marcotte R, Ferrer G, Olson E. ApriISAM: real-time smoothing and mapping. In: 2018 IEEE International Conference on Robotics and Automation (ICRA).risbane, QLD. New York, USA: IEEE, 2018 DOI:10.1109/icra.2018.8461072

67. Kaess M, Johannsson H, Roberts R, Ila V, Leonard J J, Dellaert F. ISAM2: Incremental smoothing and mapping using the Bayes tree. The International Journal of Robotics Research, 2012, 31(2): 216–235 DOI:10.1177/0278364911430419

68. Ila V, Polok L, Solony M, Svoboda P. SLAM++-A highly efficient and temporally scalable incremental SLAM framework. The International Journal of Robotics Research, 2017, 36(2): 210–230 DOI:10.1177/0278364917691110

69. Hinton G E. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504–507 DOI:10.1126/science.1127647

70. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436–444 DOI:10.1038/nature14539

71. Detone D, Malisiewicz T, Rabinovich A. Toward geometric deep SLAM. arXiv preprint arXiv:1707.07410, 2017

72. Tang J X, Ericson L, Folkesson J, Jensfelt P. GCNv2: efficient correspondence prediction for real-time SLAM. IEEE Robotics and Automation Letters, 2019: 1 DOI:10.1109/lra.2019.2927954

73. Zeng A, Song S R, NieBner M, Fisher M, Xiao J X, Funkhouser T. 3DMatch: learning local geometric descriptors from RGB-D reconstructions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.29

74. Yi K M, Trulls E, Lepetit V, Fua P. LIFT: learned invariant feature transform// Computer Vision – ECCV 2016. Cham: Springer International Publishing, 2016, 467–483 DOI:10.1007/978-3-319-46466-4_28

75. DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: self-supervised interest point detection and description. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvprw.2018.00060

76. Verdie Y, Yi K M, Fua P, Lepetit V. TILDE: A temporally invariant learned DEtector. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015 DOI:10.1109/cvpr.2015.7299165

77. Jayaraman D, Grauman K. Learning image representations tied to ego-motion. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, IEEE, 2015 DOI:10.1109/iccv.2015.166

78. Agrawal P, Carreira J, Malik J. Learning to see by moving. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, IEEE, 2015 DOI:10.1109/iccv.2015.13

79. Schmidt U, Roth S. Learning rotation-aware features: From invariant priors to equivariant descriptors. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA, IEEE, 2012. DOI:10.1109/cvpr.2012.6247909

80. Lenc K, Vedaldi A. Understanding image representations by measuring their equivariance and equivalence. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015 DOI:10.1109/cvpr.2015.7298701

81. Luo W, Li Y, Urtasun R, Zemel R. Understanding the effective receptive field in deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS). 2016, 4898–4906

82. Lianos K N, Schönberger J L, Pollefeys M, Sattler T. VSO: visual semantic odometry// Computer Vision―ECCV 2018. Cham: Springer International Publishing, 2018, 246–263 DOI:10.1007/978-3-030-01225-0_15

83. Barsan I A, Liu P D, Pollefeys M, Geiger A. Robust dense mapping for large-scale dynamic environments. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane, Australia, IEEE, 2018 DOI:10.1109/icra.2018.8462974

84. Sunderhauf N, Pham T T, Latif Y, Milford M, Reid I. Meaningful maps with object-oriented semantic mapping. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, Canada, IEEE, 2017 DOI:10.1109/iros.2017.8206392

85. Dame A, Prisacariu V A, Ren C Y, Reid I. Dense reconstruction using 3D object shape priors. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, IEEE, 2013 DOI:10.1109/cvpr.2013.170

86. Salas-Moreno R F, Newcombe R A, Strasdat H, Kelly P H J, Davison A J. SLAM++: simultaneous localisation and mapping at the level of objects. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, IEEE, 2013 DOI:10.1109/cvpr.2013.178

87. McCormac J, Handa A, Davison A, Leutenegger S. SemanticFusion: Dense 3D semantic mapping with convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore, IEEE, 2017 DOI:10.1109/icra.2017.7989538

88. Eigen D, Puhrsch C, Fergus R. Depth map prediction from a single image using a multi-scale deep network. Proceedings of the 27th International Conference on Neural Information Processing Systems,2014, 2: 2366–2374

89. Tateno K, Tombari F, Laina I, Navab N. CNN-SLAM: real-time dense monocular SLAM with learned depth prediction. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.695

90. Yin X C, Wang X W, Du X G, Chen Q J. Scale recovery for monocular visual odometry using depth estimated with deep convolutional neural Fields. In: 2017 IEEE International Conference on Computer Vision (ICCV. Venice, Italy, IEEE, 2017 DOI:10.1109/iccv.2017.625

91. Tang J X, Folkesson J, Jensfelt P. Sparse2Dense: from direct sparse odometry to dense 3-D reconstruction. IEEE Robotics and Automation Letters, 2019, 4(2): 530–537 DOI:10.1109/lra.2019.2891433

92. Bloesch M, Czarnowski J, Clark R, Leutenegger S, Davison A J. CodeSLAM―learning a compact, optimisable representation for dense visual SLAM. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00271

93. Zhou T H, Brown M, Snavely N, Lowe D G. Unsupervised learning of depth and ego-motion from video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.700

94. Zhan H Y, Garg R, Weerasekera C S, Li K J, Agarwal H, Reid I M. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00043

95. Li R H, Wang S, Long Z Q, Gu D B. UnDeepVO: monocular visual odometry through unsupervised deep learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, USA, IEEE, 2018 DOI:10.1109/icra.2018.8461251

96. Mahjourian R, Wicke M, Angelova A. Unsupervised learning of depth and ego-motion from monocular video using 3D geometric constraints. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00594

97. Yin Z C, Shi J P. GeoNet: unsupervised learning of dense depth, optical flow and camera pose. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00212

98. Madhu Babu V, Das K, Majumdar A, Kumar S. UnDEMoN: unsupervised deep network for depth and ego-motion estimation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain, USA, IEEE, 2018 DOI:10.1109/iros.2018.8593864

99. Almalioglu Y, Saputra M R U, de Gusmao P P B, Markham A, Trigoni N. GANVO: unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks. In: 2019 International Conference on Robotics and Automation (ICRA). Montreal, QC, Canada, IEEE, 2019 DOI:10.1109/icra.2019.8793512

100. Costante G, Mancini M, Valigi P, Ciarfuglia T A. Exploring representation learning with CNNs for frame-to-frame ego-motion estimation. IEEE Robotics and Automation Letters, 2016, 1(1): 18–25 DOI:10.1109/lra.2015.2505717

101. Wang S, Clark R, Wen H K, Trigoni N. DeepVO: Towards end-to-end visual odometry with deep Recurrent Convolutional Neural Networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore, IEEE, 2017 DOI:10.1109/icra.2017.7989236

102. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T. FlowNet: learning optical flow with convolutional networks. In: 2015 IEEE International Conference on Computer Vision. Santiago, Chile, IEEE, 2015 DOI:10.1109/iccv.2015.316

103. Wang S, Clark R, Wen H K, Trigoni N. End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks. The International Journal of Robotics Research, 2018, 37(4/5): 513–542 DOI:10.1177/0278364917734298

104. Ummenhofer B, Zhou H Z, Uhrig J, Mayer N, Ilg E, Dosovitskiy A, Brox T. DeMoN: depth and motion network for learning monocular stereo. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)．Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.596

105. Zhou H Z, Ummenhofer B, Brox T. DeepTAM: deep tracking and mapping// Computer Vision – ECCV 2018. Cham: Springer International Publishing, 2018, 851–868 DOI:10.1007/978-3-030-01270-0_50

106. Xue F, Wang Q Y, Wang X, Dong W, Wang J Q, Zha H B. Guided feature selection for deep visual odometry// Computer Vision―ACCV 2018. Cham: Springer International Publishing, 2019, 293–308 DOI:10.1007/978-3-030-20876-9_19

107. Zhao C, Sun L, Purkait P, Duckett T, Stolkin R. Learning monocular visual odometry with dense 3D mapping from dense 3D flow. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain, IEEE, 2018 DOI:10.1109/iros.2018.8594151

108. Xue F, Wang X, Li S, Wang Q, Wang J, Zha H. Beyond tracking: selecting memory and refining poses for deep visual odometry. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 8575–8583

109. Kendall A, Grimes M, Cipolla R. PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, IEEE, 2015 DOI:10.1109/iccv.2015.336

110. Brahmbhatt S, Gu J W, Kim K, Hays J, Kautz J. Geometry-aware learning of maps for camera localization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00277

111. Sattler T, Leibe B, Kobbelt L. Efficient & effective prioritized matching for large-scale image-based localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(9): 1744–1756 DOI:10.1109/tpami.2016.2611662

112. Sattler T, Zhou Q, Pollefeys M, Leal-Taixe L. Understanding the limitations of CNN-based absolute camera pose regression. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019, 3302–3312

113. Lai H, Tsai Y, Chiu W. Bridging Stereo Matching and Optical Flow via Spatiotemporal Correspondence. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019, 1890–1899

114. Kendall A, Cipolla R. Modelling uncertainty in deep learning for camera relocalization. In: 2016 IEEE International Conference on Robotics and Automation (ICRA). Stockholm, Sweden, IEEE, 2016 DOI:10.1109/icra.2016.7487679

115. Kendall A, Cipolla R. Geometric loss functions for camera pose regression with deep learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.694

116. Walch F, Hazirbas C, Leal-Taixe L, Sattler T, Hilsenbeck S, Cremers D. Image-based localization using LSTMs for structured feature correlation. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy, IEEE, 2017 DOI:10.1109/iccv.2017.75

117. Clark R, Wang S, Markham A, Trigoni N, Wen H K. VidLoc: A deep spatio-temporal model for 6-DoF video-clip relocalization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, Italy, IEEE, 2017 DOI:10.1109/cvpr.2017.284

118. Mourikis A I, Roumeliotis S I. A multi-state constraint kalman filter for vision-aided inertial navigation. In: Proceedings 2007 IEEE International Conference on Robotics and Automation. Rome, Italy, IEEE, 2007 DOI:10.1109/robot.2007.364024

119. Usenko V, Engel J, Stuckler J, Cremers D. Direct visual-inertial odometry with stereo cameras. In: 2016 IEEE International Conference on Robotics and Automation (ICRA). Stockholm, Sweden, 2016 DOI:10.1109/icra.2016.7487335

120. von Stumberg L, Usenko V, Cremers D. Direct sparse visual-inertial odometry using dynamic marginalization. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane, Australia, IEEE, 2018 DOI:10.1109/icra.2018.8462905

121. Bloesch M, Burri M, Omari S, Hutter M, Siegwart R. Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback. The International Journal of Robotics Research, 2017, 36(10): 1053–1072 DOI:10.1177/0278364917728574

122. Leutenegger S, Lynen S, Bosse M, Siegwart R, Furgale P. Keyframe-based visual–inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 2015, 34(3): 314–334 DOI:10.1177/0278364914554813

123. Mur-Artal R, Tardos J D. Visual-inertial monocular SLAM with map reuse. IEEE Robotics and Automation Letters, 2017, 2(2): 796–803 DOI:10.1109/lra.2017.2653359

124. Weikersdorfer D, Hoffmann R, Conradt J. Simultaneous localization and mapping for event-based vision systems// Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, 133–142 DOI:10.1007/978-3-642-39402-7_14

125. Brandli C, Berner R, Yang M H, Liu S C, Delbruck T. A 240 × 180 130 dB 3 µs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 2014, 49(10): 2333–2341 DOI:10.1109/jssc.2014.2342715

126. Kueng B, Mueggler E, Gallego G, Scaramuzza D. Low-latency visual odometry using event-based feature tracks. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon, SouthKorea, IEEE, 2016 DOI:10.1109/iros.2016.7758089

127. Kim H, Leutenegger S, Davison A J. Real-time 3D reconstruction and 6-DoF tracking with an event camera// Computer Vision―ECCV 2016. Cham: Springer International Publishing, 2016, 349–364 DOI:10.1007/978-3-319-46466-4_21

128. Rebecq H, Horstschaefer T, Gallego G, Scaramuzza D. EVO: A geometric approach to event-based 6-DOF parallel tracking and mapping in real time. IEEE Robotics and Automation Letters, 2017, 2(2): 593–600 DOI:10.1109/lra.2016.2645143

129. Weikersdorfer D, Adrian D B, Cremers D, Conradt J. Event-based 3D SLAM with a depth-augmented dynamic vision sensor. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). HongKong, China, IEEE, 2014 DOI:10.1109/icra.2014.6906882

130. Milford M, Kim H, Leutenegger S, Davison A. Towards visual SLAM with event-based cameras place recognition on event data using SeqSLAM. In: The problem of mobile sensors workshop in conjunction with RSS. 2015

131. Ovren H, Forssen P E. Spline error weighting for robust visual-inertial fusion. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018 DOI:10.1109/cvpr.2018.00041

132. Mueggler E, Gallego G, Rebecq H, Scaramuzza D. Continuous-time visual-inertial odometry for event cameras. IEEE Transactions on Robotics, 2018, 34(6): 1425–1440 DOI:10.1109/tro.2018.2858287

133. Anderson S, Barfoot T D. Towards relative continuous-time SLAM. In: 2013 IEEE International Conference on Robotics and Automation. Karlsruhe, Germany, IEEE, 2013 DOI:10.1109/icra.2013.6630700

134. Kerl C, Stuckler J, Cremers D. Dense continuous-time tracking and mapping with rolling shutter RGB-D cameras. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, IEEE, 2015 DOI:10.1109/iccv.2015.261

135. Furgale P, Tong C H, Barfoot T D, Sibley G. Continuous-time batch trajectory estimation using temporal basis functions. The International Journal of Robotics Research, 2015, 34(14): 1688–1710 DOI:10.1177/0278364915585860

136. Lovegrove S, Patron-Perez A, Sibley G. Spline Fusion: A continuous-time representation for visual-inertial fusion with application to rolling shutter cameras. In: Procedings of the British Machine Vision Conference 2013. Bristol, UK, 2013 DOI:10.5244/c.27.93

137. Tong C H, Furgale P, Barfoot T D. Gaussian Process Gauss–Newton for non-parametric simultaneous localization and mapping. The International Journal of Robotics Research, 2013, 32(5): 507–525 DOI:10.1177/0278364913478672

138. Anderson S, Dellaert F, Barfoot T D. A hierarchical wavelet decomposition for continuous-time SLAM. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). HongKong, China, IEEE, 2014 DOI:10.1109/icra.2014.6906884

139. Anderson S, Barfoot T D, Tong C H, Särkkä S. Batch nonlinear continuous-time trajectory estimation as exactly sparse Gaussian process regression. Autonomous Robots, 2015, 39(3): 221–238 DOI:10.1007/s10514-015-9455-y

140. Barfoot T, Hay Tong C, Sarkka S. Batch continuous-time trajectory estimation as exactly sparse Gaussian process regression. In: Robotics: Science and Systems X, Robotics: Science and Systems Foundation, 2014 DOI:10.15607/rss.2014.x.001

141. Yan X Y, Indelman V, Boots B. Incremental sparse GP regression for continuous-time trajectory estimation and mapping. Robotics and Autonomous Systems, 2017, 87, 120–132 DOI:10.1016/j.robot.2016.10.004

142. Anderson S, Barfoot T D. Full STEAM ahead: Exactly sparse Gaussian process regression for batch continuous-time trajectory estimation on SE(3). In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany, IEEE, 2015 DOI:10.1109/iros.2015.7353368

2. Rafik HAMZA, Minh-Son DAO, Privacy-preserving deep learning techniques for wearable sensor-based big data applications Virtual Reality & Intelligent Hardware 2022, 4(3): 210-222

3. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246

4. Lianyu ZHENG, Xinyu LIU, Zewu AN, Shufei LI, Renjie ZHANG, A smart assistance system for cable assembly by combining wearable augmented reality with portable visual inspection Virtual Reality & Intelligent Hardware 2020, 2(1): 12-27

5. Zhiyuan ZHANG, Yuchao DAI, Jiadai SUN, Deep learning based point cloud registration: an overview Virtual Reality & Intelligent Hardware 2020, 2(3): 222-246

6. Muhammad IRFAN, Muhammad MUNSIF, Deepdive: a learning-based approach for virtual camera in immersive contents Virtual Reality & Intelligent Hardware 2022, 4(3): 247-262

7. Yuanyuan SHI, Yunan LI, Xiaolong FU, Kaibin MIAO, Qiguang MIAO, Review of dynamic gesture recognition Virtual Reality & Intelligent Hardware 2021, 3(3): 183-206

8. Xiaojiao SONG, Jianjun ZHU, Jingfan FAN, Danni AI, Jian YANG, Topological distance-constrained feature descriptor learning model for vessel matching in coronary angiographies Virtual Reality & Intelligent Hardware 2021, 3(4): 287-301

9. Wei LYU, Zhong ZHOU, Lang CHEN, Yi ZHOU, A survey on image and video stitching Virtual Reality & Intelligent Hardware 2019, 1(1): 55-83

10. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

11. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

12. Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG, Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems Virtual Reality & Intelligent Hardware 2021, 3(4): 274-286

13. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

14. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

15. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

16. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

17. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

18. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

19. Pengfei HAN, Gang ZHAO, A review of edge-based 3D tracking of rigid objects Virtual Reality & Intelligent Hardware 2019, 1(6): 580-596

20. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

21. Lingfei ZHU, Qi CAO, Yiyu CAI, Development of augmented reality serious games with a vibrotactile feedback jacket Virtual Reality & Intelligent Hardware 2020, 2(5): 454-470

22. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

23. Jie REN, Chun YU, Yueting WENG, Chengchi ZHOU, Yuanchun SHI, Design and evaluation of window management operations in AR headset+smartphone interface Virtual Reality & Intelligent Hardware 2022, 4(2): 115-131

24. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

25. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

26. Ahmed L ALYOUSIFY, Ramadhan J MSTAFA, AR-assisted children book for smart teaching and learning of Turkish alphabets Virtual Reality & Intelligent Hardware 2022, 4(3): 263-277