A review of edge-based 3D tracking of rigid objects

Abstract

Three-dimensional (3D) tracking of rigid objects plays a very important role in many areas such as augmented reality, computer vision, and robotics. Numerous works have been done to pursue more stable, faster, and more accurate 3D tracking. Among various tracking methods, edge-based 3D tracking has been widely used owing to its many advantages. Furthermore, edge-based methods can be mainly divided into two categories, methods without and those with explicit edges, depending on whether explicit edges need to be extracted. Based on this, representative methods in both categories are introduced, analyzed, and compared in this paper. Finally, some suggestions on the choice of methods in different application scenarios and research directions in the future are given.

Keyword

Augmented reality ; 3D tracking ; Edge ; CAD model

Cite this article

Pengfei HAN, Gang ZHAO. A review of edge-based 3D tracking of rigid objects. Virtual Reality & Intelligent Hardware, 2019, 1(6): 580-596 DOI:10.1016/j.vrih.2019.10.001

References

1. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 DOI:10.1023/b:visi.0000029664.99615.94

2. Bay H, Ess A, Tuytelaars T, van Gool L. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346–359 DOI:10.1016/j.cviu.2007.09.014

3. Deriche R, Giraudon G. A computational approach for corner and vertex detection. International Journal of Computer Vision, 1993, 10(2): 101–124 DOI:10.1007/bf01420733

4. Smith S M and Brady J M. SUSAN-A New approach to low level image processing. International Journal of Computer Vision, 1997, 23(1): 45–78 DOI:10.1023/a:1007963824710

5. Horn B K P, Schunck B G. Determining optical flow. Artificial Intelligence, 1981, 17(1/2/3): 185–203 DOI:10.1016/0004-3702(81)90024-2

6. Lepetit V, Fua P. Monocular model-based 3D tracking of rigid objects: A survey. Foundations and Trends® in Computer Graphics and Vision, 2005, 1(1): 1–89 DOI:10.1561/0600000001

7. Harris C and Stennett C. RAPID-a video rate object tracker. BMVC. 1990: 1–6 DOI:10.5244/C.4.15

8. Floudas C A and Pardalos P M. Encyclopedia of optimization. Reference Reviews, 2009, 584(4): 31–52 DOI:10.1016/0375-6505(85)90011-2

9. Levenberg K. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 1944, 2(2): 164–168 DOI:10.1090/qam/10666

10. Powell M J D. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The Computer Journal, 1964, 7(2): 155–162 DOI:10.1093/comjnl/7.2.155

11. Simon G, Berger M O. A two-stage robust statistical method for temporal registration from features of various type. In: Sixth International Conference on Computer Vision. IEEE, 1998, 261–266 DOI:10.1109/iccv.1998.710728

12. Drummond T, Cipolla R. Real-time visual tracking of complex structures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 932–946 DOI:10.1109/tpami.2002.1017620

13. MarchandÉ, Bouthemy P, Chaumette F. A 2D–3D model-based approach to real-time visual tracking. Image and Vision Computing, 2001, 19(13): 941–955 DOI:10.1016/s0262-8856(01)00054-3

14. Vacchetti L, Lepetit V, Fua P. Combining edge and texture information for real-time accurate 3D camera tracking. In: Third IEEE and ACM International Symposium on Mixed and Augmented Reality Arlington. VA, USA, 2004, 48–56 DOI:10.1109/ismar.2004.24

15. Wuest H, Vial F, Strieker D. Adaptive line tracking with multiple hypotheses for augmented reality. In: Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05). Vienna, Austria, IEEE, 2005DOI:10.1109/ismar.2005.8

16. Armstrong M, Zisserman A. Robust object tracking. Proceedings of the Asian Conference on Computer Vision, 1995: 5–8

17. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

18. Wuest H, Stricker D. Tracking of industrial objects by using cad models. JVRB-Journal of Virtual Reality and Broadcasting, 2007, 4(1) DOI:10.20385/1860-2037/4.2007.1

19. Wang B, Zhong F, Qin X Y. Robust edge-based 3D object tracking with direction-based pose validation. Multimedia Tools and Applications, 2019, 78(9): 12307–12331 DOI:10.1007/s11042-018-6727-5

20. Moral P D. Nonlinear filtering: Interacting particle resolution. Markov Processes and Related Fields, 1996, 2 (4): 555–580 DOI:10.1016/s0764-4442(97)84778-7

21. DeCarlo D, Metaxas D. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In: Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA, USA, 1996DOI:10.1109/cvpr.1996.517079

22. Jurie F, Dhome M. Hyperplane approximation for template matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 996–1000 DOI:10.1109/tpami.2002.1017625

23. Masson L, Jurie F, Dhome M. Contour/texture approach for visual tracking//Image Analysis. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003: 661–668 DOI:10.1007/3-540-45103-x_88

24. Bugaev B, Kryshchenko A, Belov R. Combining 3D model contour energy and keypoints for object tracking// Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018: 55–70 DOI:10.1007/978-3-030-01258-8_4

25. Rublee E, Rabaud V, Konolige K, Bradski G. ORB: An efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011DOI:10.1109/iccv.2011.6126544

26. Calonder M, Lepetit V, Strecha C, Fua P. BRIEF: binary robust independent elementary features//Computer Vision–ECCV 2010. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010: 778–792 DOI:10.1007/978-3-642-15561-1_56

27. Leutenegger S, Chli M, Siegwart R Y. BRISK: Binary Robust invariant scalable keypoints. In: 2011 International Conference on Computer Vision. Barcelona, Spain, IEEE, 2011, 2548–2555 DOI:10.1109/iccv.2011.6126542

28. Alahi A, Ortiz R, Vandergheynst P. FREAK: fast retina keypoint. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, IEEE, 2012, 510–517 DOI:10.1109/cvpr.2012.6247715

29. Seo B K, Park J, Park H, Park J I. Real-time visual tracking of less textured three-dimensional objects on mobile platforms. Optical Engineering, 2013, 51(12): 127202 DOI:10.1117/1.oe.51.12.127202

30. Choi C, Christensen H I. Robust 3D visual tracking using particle filtering on the special Euclidean group: A combined approach of keypoint and edge features. The International Journal of Robotics Research, 2012, 31(4): 498–519 DOI:10.1177/0278364912437213

31. Metaio: Edge-based initialization (2015). http://www.metaio.com/

32. Vuforia Engine (2018). https://developer.vuforia.com/

33. VisionLib (2019). https://visionlib.com/

34. Han P. Edge-based real-time tracking for mobile augmented reality on iphone x. https://v.youku.com/v_show/id_XNDA5MTQzNTUwOA==.html?spm=a2h3j.8428770.3416059.1.

35. Roller D, Daniilidis K, Nagel H H. Model-based object tracking in monocular image sequences of road traffic scenes. International Journal of Computer Vision, 1993, 10(3): 257–281 DOI:10.1007/bf01539538

36. Deriche R, Faugeras O. Tracking line segments. Image and Vision Computing, 1990, 8(4): 261–270 DOI:10.1016/0262-8856(90)80002-b

37. Shahrokni A, Vacchetti L, Lepetit V, Fua P. Polyhedral object detection and pose estimation for augmented reality applications. In Proceedings of Computer Animation2002. Geneva, Switzerland, IEEE Comput. Soc, 2002, 65–69 DOI:10.1109/ca.2002.1017508

38. David P, DeMenthon D. Object recognition in high clutter images using line features. In: Tenth IEEE International Conference on Computer Vision. Beijing, China, IEEE, 2005, 1581–1588 DOI:10.1109/iccv.2005.173

39. Kotake D, Satoh K, Uchiyama S, Yamamoto H. A fast initialization method for edge-based registration using an inclination constraint. In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan, IEEE, 2007, 239–248 DOI:10.1109/ismar.2007.4538854

40. Kim G, Hebert M, Park S K. Preliminary development of a line feature-based object recognition system for textureless indoor objects//Recent Progress in Robotics: Viable Robotic Service to Human. Berlin, Heidelberg, 2007, 255–268 DOI:10.1007/978-3-540-76729-9_20

41. Lu Z, Li Y, Wang J, et al. 3D Object Recognition Using Line Structure. Proceedings of SPIE-The International Society for Optical Engineering, 2011, 8009(4):41 DOI:10.1117/12.896087

42. Álvarez H, Borro D. Junction assisted 3D pose retrieval of untextured 3D models in monocular images. Computer Vision and Image Understanding, 2013, 117(10): 1204–1214 DOI:10.1016/j.cviu.2012.08.012

43. Qiu Z Y, Wei H. Line segment based man-made object recognition using invariance. In: 2009 International Joint Conference on Artificial Intelligence. Hainan Island, China, IEEE, 2009, 460–464 DOI:10.1109/jcai.2009.149

44. Rockafellar R T, Wets R J B. Variational analysis. Springer Science & Business Media, 2009, 317 DOI:10.1007/978-3-642-02431-3

45. Guerra C, Pascucci V. Line-based object recognition using Hausdorff distance: from range images to molecular secondary structures. Image and Vision Computing, 2005, 23(4): 405–415 DOI:10.1016/j.imavis.2004.11.002

46. Glover F, Laguna M. Tabu search//Handbook of Combinatorial Optimization. Boston, MA: Springer US, 1998, 2093–2229 DOI:10.1007/978-1-4613-0303-9_33

47. Han P F, Zhao G. CAD-based 3D objects recognition in monocular images for mobile augmented reality. Computers & Graphics, 2015, 50: 36–46 DOI:10.1016/j.cag.2015.05.021

48. Han P F, Zhao G. Line-based initialization method for mobile augmented reality in aircraft assembly. The Visual Computer, 2017, 33(9): 1185–1196 DOI:10.1007/s00371-016-1281-5

49. Durrant-Whyte H, Bailey T. Simultaneous localization and mapping: part I. IEEE Robotics & Automation Magazine, 2006, 13(2): 99–110 DOI:10.1109/mra.2006.1638022

50. Nister D, Naroditsky O, Bergen J. Visual odometry. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC, USA, IEEE, 2004, 1: I–I DOI:10.1109/cvpr.2004.1315094

51. Brunetti A, Buongiorno D, Trotta G F, Bevilacqua V. Computer vision and deep learning techniques for pedestrian detection and tracking: A survey. Neurocomputing, 2018, 300: 17–33 DOI:10.1016/j.neucom.2018.01.092

52. Garon M, Lalonde J F. Deep 6-DOF tracking. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(11): 2410–2418 DOI:10.1109/tvcg.2017.2734599

53. Schulter S, Vernaza P, Choi W, Chandraker M. Deep network flow for multi-object tracking. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, 2017: 6951–6960 DOI:10.1109/cvpr.2017.292

54. Wang N, Yeung D Y. Learning a deep compact image representation for visual tracking. Advances in neural information processing systems. 2013: 809–817 DOI:http://respository.ust.hk/ir/Record/1783.1-61168

55. Nam H, Han B. Learning multi-domain convolutional neural networks for visual tracking. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016, 4293–4302 DOI:10.1109/cvpr.2016.465

56. Scheidegger S, Benjaminsson J, Rosenberg E, Krishnan A, Granstrom K. Mono-camera 3D multi-object tracking using deep learning detections and PMBM filtering. In: 2018 IEEE Intelligent Vehicles Symposium (IV). Changshu, IEEE, 2018, 433–440 DOI:10.1109/ivs.2018.8500454

57. Pauwels K, Ivan V, Ros E, Vijayakumar S. Real-time object pose recognition and tracking with an imprecisely calibrated moving RGB-D camera. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Chicago, IL, USA, IEEE, 2014, 2733–2740 DOI:10.1109/iros.2014.6942936

58. Petit A, Lippiello V, Siciliano B. Real-time tracking of 3D elastic objects with an RGB-D sensor. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany, IEEE, 2015, 3914–3921 DOI:10.1109/iros.2015.7353928

59. Choi C, Christensen H I. RGB-D object tracking: A particle filter approach on GPU. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo, New York, IEEE, 2013, 1084–1091 DOI:10.1109/iros.2013.6696485

60. Park Y, Lepetit V, Woo W. Texture-less object tracking with online training using an RGB-D camera. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality. Basel, New York, 2011, 121–126 DOI:10.1109/ismar.2011.6162879

2. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

3. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

4. Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG, Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems Virtual Reality & Intelligent Hardware 2021, 3(4): 274-286

5. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

6. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

7. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

8. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

9. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

10. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

11. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

12. Lingfei ZHU, Qi CAO, Yiyu CAI, Development of augmented reality serious games with a vibrotactile feedback jacket Virtual Reality & Intelligent Hardware 2020, 2(5): 454-470

13. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

14. Jie REN, Chun YU, Yueting WENG, Chengchi ZHOU, Yuanchun SHI, Design and evaluation of window management operations in AR headset+smartphone interface Virtual Reality & Intelligent Hardware 2022, 4(2): 115-131

15. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

16. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460

17. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

18. Rafik HAMZA, Minh-Son DAO, Privacy-preserving deep learning techniques for wearable sensor-based big data applications Virtual Reality & Intelligent Hardware 2022, 4(3): 210-222

19. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246

20. Ahmed L ALYOUSIFY, Ramadhan J MSTAFA, AR-assisted children book for smart teaching and learning of Turkish alphabets Virtual Reality & Intelligent Hardware 2022, 4(3): 263-277