Collaborative visual SLAM for multiple agents: A brief survey

Abstract

This article presents a brief survey to visual simultaneous localization and mapping (SLAM) systems applied to multiple independently moving agents, such as a team of ground or aerial vehicles, a group of users holding augmented or virtual reality devices. Such visual SLAM system, name as collaborative visual SLAM, is different from a typical visual SLAM deployed on a single agent in that information is exchanged or shared among different agents to achieve better robustness, efficiency, and accuracy. We review the representative works on this topic proposed in the past ten years and describe the key components involved in designing such a system including collaborative pose estimation and mapping tasks, as well as the emerging topic of decentralized architecture. We believe this brief survey could be helpful to someone who are working on this topic or developing multi-agent applications, particularly micro-aerial vehicle swarm or collaborative augmented/virtual reality.

Keyword

Visual SLAM ; Multiple agent ; UAV swarm ; Collaborative AR/VR

Cite this article

Danping ZOU, Ping TAN, Wenxian YU. Collaborative visual SLAM for multiple agents: A brief survey. Virtual Reality & Intelligent Hardware, 2019, 1(5): 461-482 DOI:10.1016/j.vrih.2019.09.002

References

1. Fuentes-Pacheco J, Ruiz-Ascencio J, Rendón-Mancha J M. Visual simultaneous localization and mapping: a survey. Artificial Intelligence Review, 2015, 43(1): 55–81 DOI:10.1007/s10462-012-9365-8

2. Scaramuzza D, Fraundorfer F. Tutorial: visual odometry. IEEE Robotics and Automation Magazine, 2011, 18(4), 80–92

3. Klinker K, Berkemeier L, Zobel B, Wüller H, Huck-Fries V, Wiesche M, HartmutRemmers, OliverThomas, Krcmar H. Structure for innovations: A use case taxonomy for smart glasses in service processes. Multikonferenz Wirtschaftsin-formatik Lüneburg, Deutschland. 2018

4. Siegwart R, Nourbakhsh I R, Scaramuzza D. Introduction to autonomous mobile robots. MIT press, 2011

5. Whyte H D. Simultaneous localisation and mapping (SLAM): Part I the essential algorithms. Robotics and Automation Magazine, 2006, 1–9

6. Bailey T, Durrant-Whyte H. Simultaneous localization and mapping (SLAM): part II. IEEE Robotics & Automation Magazine, 2006, 13(3): 108–117 DOI:10.1109/mra.2006.1678144

7. Schonberger J L, Frahm J M. Structure-from-motion revisited. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016, 4104–4113 DOI:10.1109/cvpr.2016.445

8. Davison A J, Reid I D, Molton N D, Stasse O. MonoSLAM: real-time single camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 1052–1067 DOI:10.1109/tpami.2007.1049

9. Welch G, Bishop G. An introduction to the Kalman Filter: Department of Computer science. University of North Carolina, 2006

10. Klein G, Murray D. Parallel tracking and mapping for small AR workspaces. In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan, IEEE, 2007, 1–10 DOI:10.1109/ismar.2007.4538852

11. Schonberger J L, Frahm J M. Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 4104–4113

12. Triggs B, McLauchlan P F, Hartley R I, Fitzgibbon A W. Bundle adjustment—A modern synthesis//Vision Algorithms: Theory and Practice. Berlin, Heidelberg: Springer, 2000, 298–372 DOI:10.1007/3-540-44480-7_21

13. Mur-Artal R, Montiel J M M, Tardos J D. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 2015, 31(5): 1147–1163 DOI:10.1109/tro.2015.2463671

14. Zou D P, Tan P. CoSLAM: collaborative visual SLAM in dynamic environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 354–366 DOI:10.1109/tpami.2012.104

15. Liu H M, Zhang G F, Bao H J. Robust keyframe-based monocular SLAM for augmented reality. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). Merida, Yucatan, Mexico, IEEE, 2016 DOI:10.1109/ismar-adjunct.2016.0111

16. Mourikis A I, Roumeliotis S I. A multi-state constraint kalman filter for vision-aided inertial navigation. In: Proceedings 2007 IEEE International Conference on Robotics and Automation. Rome, Italy, IEEE, 2007, 3565–3572 DOI:10.1109/robot.2007.364024

17. Li M Y, Mourikis A I. High-precision, consistent EKF-based visual-inertial odometry. The International Journal of Robotics Research, 2013, 32(6): 690–711 DOI:10.1177/0278364913481251

18. Triggs B, McLauchlan P F, Hartley R I, Fitzgibbon A W. Bundle adjustment—A modern synthesis//Vision Algorithms: Theory and Practice. Berlin, Heidelberg: Springer, 2000, 298–372 DOI:10.1007/3-540-44480-7_21

19. Kaess M, Johannsson H, Roberts R, Ila V, Leonard J J, Dellaert F. ISAM2: Incremental smoothing and mapping using the Bayes tree. The International Journal of Robotics Research, 2012, 31(2): 216–235 DOI:10.1177/0278364911430419

20. Galvez-López D, Tardos J D. Bags of binary words for fast place recognition in image sequences. IEEE Transactions on Robotics, 2012, 28(5): 1188–1197 DOI:10.1109/tro.2012.2197158

21. Straub J, Hilsenbeck S, Schroth G, Huitl R, Möller A, Steinbach E. Fast relocalization for visual odometry using binary features. In: 2013 IEEE International Conference on Image Processing. Melbourne, VIC, Australia, 2013, IEEE, 2548–2552

22. Williams B, Klein G, Reid I. Real-time SLAM relocalisation. In: IEEE 11th International Conference on. Rio de Janeiro, Brazil, IEEE, 2007, 1–8

23. Newcombe R A, Lovegrove S J, Davison A J. DTAM: Dense tracking and mapping in real-time. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011 DOI:10.1109/iccv.2011.6126513

24. Engel J, Schops T, Cremers D. LSD-SLAM: Direct Monocular SLAM. Computer Vision–ECCV, 2014

25. Engel J, Koltun V, Cremers D. Direct Sparse Odometry. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(3), 611–625

26. Forster C, Pizzoli M, Scaramuzza D. SVO: Fast semi-direct monocular visual odometry. In: 2014 IEEE International Conference on Robotics and Automation. Hong Kong, China, IEEE, 2014 DOI:10.1109/icra.2014.6906584

27. Forster C, Zhang Z C, Gassner M, Werlberger M, Scaramuzza D. SVO: semidirect visual odometry for monocular and multicamera systems. IEEE Transactions on Robotics, 2017, 33(2): 249–265 DOI:10.1109/tro.2016.2623335

28. Leutenegger S, Lynen S, Bosse M, Siegwart R, Furgale P. Keyframe-based visual–inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 2015, 34(3): 314–334 DOI:10.1177/0278364914554813

29. Qin T, Li P L, Shen S J. VINS-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018, 34(4): 1004–1020 DOI:10.1109/tro.2018.2853729

30. Zou D P, Wu Y X, Pei L, Ling H B, Yu W X. StructVIO: visual-inertial odometry with structural regularity of man-made environments. IEEE Transactions on Robotics, 2019, 35(4): 999–1013 DOI:10.1109/tro.2019.2915140

31. Liu H M, Chen M Y, Zhang G F, Bao H J, Bao Y Z. ICE-BA: incremental, consistent and efficient bundle adjustment for visual-inertial SLAM. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 1974–1982 DOI:10.1109/cvpr.2018.00211

32. Shafi M, Molisch A F, Smith P J, Haustein T, Zhu P Y, de Silva P, Tufvesson F, Benjebbour A, Wunder G. 5G: A tutorial overview of standards, trials, challenges, deployment, and practice. IEEE Journal on Selected Areas in Communications, 2017, 35(6): 1201–1221 DOI:10.1109/jsac.2017.2692307

33. Howard A. Multi-robot simultaneous localization and mapping using particle filters. The International Journal of Robotics Research, 2006, 25(12): 1243–1256 DOI:10.1177/0278364906072250

34. Kim B, Kaess M, Fletcher L, Leonard J, Bachrach A, Roy N, Teller S. Multiple relative pose graphs for robust cooperative mapping. In: 2010 IEEE International Conference on Robotics and Automation. Anchorage, AK, USA, IEEE, 2010 DOI:10.1109/robot.2010.5509154

35. Schmuck P, Chli M. CCM-SLAM: Robust and efficient centralized collaborative monocular simultaneous localization and mapping for robotic teams. Journal of Field Robotics, 2019, 36(4): 763–781 DOI:10.1002/rob.21854

36. Karrer M, Schmuck P, Chli M. CVI-SLAM: collaborative visual-inertial SLAM. IEEE Robotics and Automation Letters, 2018, 3(4): 2762–2769 DOI:10.1109/lra.2018.2837226

37. Castle R, Klein G, Murray D W. Video-rate localization in multiple maps for wearable augmented reality. In: 2008 12th IEEE International Symposium on Wearable Computers. Pittaburgh, PA, USA, IEEE, 2008 DOI:10.1109/iswc.2008.4911577

38. Agarwal S, Snavely N, Seitz S, Szeliski R. Bundle adjustment in the large. In: Computer Vision—ECCV. Berlin, Heidelberg, Springer, 2010

39. Perron J M, HuangR, ThomasJ, Zhang L, Tan P, Vaughan R T. Orbiting a moving target with multi-robot collaborative visual slam. In: Workshop on Multi-View Geometry in Robotics. 2015, 1339–1344

40. Forster C, Lynen S, Kneip L, Scaramuzza D. Collaborative monocular SLAM with multiple Micro Aerial Vehicles. In: IEEE International Conference on Intelligent Robots and Systems. Tokyo, Japan, IEEE, 2013, 3963–3970

41. Kneip L, Scaramuzza D, Siegwart R. Robust Real-Time Visual Odometry with a Single Camera and an IMU. In: Proceedings of the British Machine Vision Conference, 2011

42. Riazuelo L, Civera J, Montiel J M M. C2TAM: A Cloud framework for cooperative tracking and mapping. Robotics and Autonomous Systems, 2014, 62(4): 401–413 DOI:10.1016/j.robot.2013.11.007

43. Civera J, Davison A J, Montiel J M M. Interacting multiple model monocular SLAM. In: 2008 IEEE International Conference on Robotics and Automation. Pasadena, CA, USA, IEEE, 2008, 3704–3709 DOI:10.1109/robot.2008.4543779

44. Loianno G, Mulgaonkar Y, Brunner C, Ahuja D, Ramanandan A, Chari M, Diaz S, Kumar V. A swarm of flying smartphones. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems. Daejeon, South Korea, IEEE, 2016, 1681–1688 DOI:10.1109/IROS.2016.7759270

45. Loianno G, Mulgaonkar Y, Brunner C, Ahuja D, Ramanandan A, Chari M, Diaz S, Kumar V. Smartphones power flying robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. Hamburg, Germany, IEEE, 2015, 1256–1263 DOI:10.1109/IROS.2015.7353530

46. Morrison J G, Gálvez-López D, Sibley G. MOARSLAM: multiple operator augmented RSLAM//Springer Tracts in Advanced Robotics. Tokyo, Japan, Springer, 2016, 119–132 DOI:10.1007/978-4-431-55879-8_9

47. Keivan N, Patron-Perez A, Sibley G. Asynchronous adaptive conditioning for visual-inertial SLAM//Experimental Robotics. Cham: Springer International Publishing, 2015, 309–321 DOI:10.1007/978-3-319-23778-7_21

48. Burri M, Nikolic J, Gohl P, Schneider T, Rehder J, Omari S, Achtelik M W, Siegwart R. The EuRoC micro aerial vehicle datasets. The International Journal of Robotics Research, 2016, 35(10): 1157–1163 DOI:10.1177/0278364915620033

49. Castle R, Klein G, Murray D W. Video-rate localization in multiple maps for wearable augmented reality. In: 2008 12th IEEE International Symposium on Wearable Computers. Pittaburgh, PA, USA, IEEE, 2008, 15–22 DOI:10.1109/iswc.2008.4911577

50. Castle R, Klein G, Murray D W. Video-rate localization in multiple maps for wearable augmented reality. In: 2008 12th IEEE International Symposium on Wearable Computers. Pittaburgh, PA, USA, IEEE, 2008 DOI:10.1109/iswc.2008.4911577

51. Davison A J, Reid I D, Molton N D, Stasse O. MonoSLAM: real-time single camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 1052–1067 DOI:10.1109/tpami.2007.1049

52. Strasdat H, Montiel J M M, Davison A J. Real-time monocular SLAM: Why filter? In: 2010 IEEE International Conference on Robotics and Automation. Anchorage, AK, USA, IEEE, 2010 DOI:10.1109/ROBOT.2010.5509636

53. Quan L, Lan Z D. Linear N-point camera pose determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1999, 21(8): 774–780 DOI:10.1109/34.784291

54. Lepetit V, Moreno-Noguer F, Fua P. EPnP: an accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 2009, 81(2): 155–166 DOI:10.1007/s11263-008-0152-6

55. Grisetti G, Kummerle R, Stachniss C, Burgard W. A Tutorial on Graph-Based SLAM. IEEE Intelligent Transportation Systems Magazine, 2010, 2(4): 31–43 DOI:10.1109/MITS.2010.939925

56. Lowry S, Sünderhauf N, Newman P, Leonard J J, Cox D, Corke P, Milford M J. Visual Place Recognition: A Survey. IEEE Transactions on Robotics, 2016, 32(1): 1–19 DOI:10.1109/TRO.2015.2496823

57. Klein G, Murray D. Improving the agility of keyframe-based SLAM//Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, 802–815 DOI:10.1007/978-3-540-88688-4_59

58. Nister D, Stewenius H. Scalable recognition with a vocabulary tree. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York, NY, USA, IEEE, 2015, 1063–6919 DOI:10.1109/cvpr.2006.264

59. Arandjelovic R, Gronat P, Torii A, Pajdla T, Sivic J. NetVLAD: CNN architecture for weakly supervised place recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, IEEE, 2016, 5297–5307 DOI:10.1109/cvpr.2016.572

60. Nistér D. An efficient solution to the five-point relative pose problem. IEEE transactions on pattern analysis and machine intelligence, 2004, 26(6), 756–777

61. Strasdat H, Davison A J, Montiel J M M, Konolige K. Double window optimisation for constant time visual SLAM. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011, 2352–2359 DOI:10.1109/iccv.2011.6126517

62. Endres F, Hess J, Engelhard N, Sturm J, Cremers D, Burgard W. An evaluation of the RGB-D SLAM system. In: 2012 IEEE International Conference on Robotics and Automation. St Paul, MN, USA, IEEE, 2012, 1691–1696 DOI:10.1109/icra.2012.6225199

63. Kummerle R, Grisetti G, Strasdat H, Konolige K, Burgard W. G2o: A general framework for graph optimization. In: 2011 IEEE International Conference on Robotics and Automation. Shanghai, China, IEEE, 2011, 3607–3613 DOI:10.1109/icra.2011.5979949

64. Mangelson J G, Dominic D, Eustice R M, Vasudevan R. Pairwise consistent measurement set maximization for robust multi-robot map merging. In: 2018 IEEE International Conference on Robotics and Automation. Brisbane, QLD, IEEE, 2018, 2916–2923 DOI:10.1109/icra.2018.8460217

65. Pattabiraman B, Patwary M M A, Gebremedhin A H, Liao W K, Choudhary A. Fast algorithms for the maximum clique problem on massive graphs with applications to overlapping community detection. Internet Mathematics, 2015, 11(4/5): 421–448 DOI:10.1080/15427951.2014.986778

66. Lourakis M I, Argyros A A. SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software, 2009, 36(1), 2 DOI:10.1145/1486525.1486527

67. Wu C, Agarwal S, Curless B, Seitz S M. Multicore bundle adjustment. In: CVPR. Providence, RI, USA, 2011, 3057–3064 DOI:10.1109/CVPR.2011.5995552

68. Agarwal S. Mierle K. Others, Ceres Solver. 2012

69. Forster C, Carlone L, Dellaert F, Scaramuzza D. On-manifold preintegration for real-time visual: inertial odometry. IEEE Transactions on Robotics, 2017, 33(1): 1–21 DOI:10.1109/tro.2016.2597321

70. Rosten E, Drummond T. Machine learning for high-speed corner detection. In: European conference on computer vision. Berlin, Heidelberg, Springer, 2006

71. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 DOI:10.1023/b:visi.0000029664.99615.94

72. Bay H, Ess A, Tuytelaars T, van Gool L. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346–359 DOI:10.1016/j.cviu.2007.09.014

73. Rublee E, Rabaud V, Konolige K, Bradski G. ORB: An efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011 DOI:10.1109/iccv.2011.6126544

74. Calonder M, Lepetit V, Strecha C, Fua P. BRIEF: Binary robust independent elementary features. In: Computer Vision―ECCV 2010. Berlin, Heidelberg, Springer, 2010, 778–792 DOI:10.1007/978-3-642-15561-1_56

75. Shi J B. Good features to track. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94. Seattle, WA, USA, IEEE, 1994, 1063–6919 DOI:10.1109/cvpr.1994.323794

76. Lucas B D, Kanade T. An iterative image registration technique with an application to stereo vision, 1981, 674–679

77. Zhou H Z, Zou D P, Pei L, Ying R D, Liu P L, Yu W X. StructSLAM: visual SLAM with building structure lines. IEEE Transactions on Vehicular Technology, 2015, 64(4): 1364–1375 DOI:10.1109/tvt.2015.2388780

78. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge: Cambridge University Press, 2004 DOI:10.1017/cbo9780511811685

79. Nistér D. Preemptive RANSAC for live structure and motion estimation. Machine Vision and Applications, 2005, 16(5): 321–329 DOI:10.1007/s00138-005-0006-y

80. Civera J, Grasa O G, Davison A J, Montiel J M M. 1-Point RANSAC for extended Kalman filtering: Application to real-time structure from motion and visual odometry. Journal of Field Robotics, 2010, 27(5): 609–631 DOI:10.1002/rob.20345

81. Wilcox R. Robust regression//Introduction to Robust Estimation and Hypothesis Testing. Elsevier, 2012, 471–532 DOI:10.1016/b978-0-12-386983-8.00010-x

82. Jiang N J, Cui Z P, Tan P. A global linear method for camera pose registration. In: 2013 IEEE International Conference on Computer Vision. Sydney, Australia, IEEE, 2013, 481–488 DOI:10.1109/iccv.2013.66

83. Cui Z P, Tan P. Global structure-from-motion by similarity averaging. In: 2015 IEEE International Conference on Computer Vision. Santiago, Chile, IEEE, 2015, 864–872 DOI:10.1109/iccv.2015.105

84. Cunningham A, Paluri M, Dellaert F. DDF-SAM: Fully distributed SLAM using constrained factor graphs. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. Taipei, Taiwan, IEEE, 2010, 3025–3030 DOI:10.1109/IROS.2010.5652875

85. Cunningham A, Indelman V, Dellaert F. DDF-SAM 2.0: Consistent distributed smoothing and mapping. In: 2013 IEEE International Conference on Robotics and Automation. Karlsruhe, Germany, IEEE, 2013, 5220–5227 DOI:10.1109/icra.2013.6631323

86. Choudhary S, Carlone L, Nieto C, Rogers J, Christensen H I, Dellaert F. Distributed mapping with privacy and communication constraints: Lightweight algorithms and object-based models. The International Journal of Robotics Research, 2017, 36(12): 1286–1311 DOI:10.1177/0278364917732640

87. Saad Y. Iterative methods for sparse linear systems. Society for Industrial and Applied Mathematics, 2003 DOI:10.1137/1.9780898718003

88. Cieslewski T, Choudhary S, Scaramuzza D. Data-efficient decentralized visual SLAM. In: 2018 IEEE International Conference on Robotics and Automation. Brisbane, QLD, Australia, IEEE, 2018, 2466–2473 DOI:10.1109/icra.2018.8461155

89. Saeedi S, Trentini M, Seto M, Li H. Multiple-robot simultaneous localization and mapping: A review. Journal of Field Robotics, 2016, 33(1): 3–46 DOI:10.1002/rob.21620

90. Chung S J, Paranjape A A, Dames P, Shen S J, Kumar V. A survey on aerial swarm robotics. IEEE Transactions on Robotics, 2018, 34(4): 837–855 DOI:10.1109/tro.2018.2857475

91. Egodagamage R, Tuceryan M. Distributed monocular visual SLAM as a basis for a collaborative augmented reality framework. Computers & Graphics, 2018, 71: 113–123 DOI:10.1016/j.cag.2018.01.002