Object registration using an RGB-D camera for complex product augmented assembly guidance

Kangkang YANG , Yu GUO , Pengzhou TANG , Haopeng ZHANG , Han LI

Department of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China

Abstract

Background

Augmented assembly guidance aims to help users complete assembly operations more efficiently and quickly through augmented reality technology, breaking the limitations of traditional assembly guidance technology which is single in content and boring in way. Object registration is one of the key technologies in augmented assembly guidance process, which can affect the location and direction of virtual assembly guidance information in real assembly environment.

Methods

This paper presents an object registration method based on RGB-D camera, which combines Lucas-Kanade (LK) optical flow algorithm and Iterative Closet Point (ICP) algorithm. An augmented assembly guidance system for complex products through this method is built. Meanwhile, in order to compare the effectiveness of the proposed method, we also implemented object registration based on an open source augmented reality SDK Vuforia.

Results

An engine model and a complex weapon cabin equipment are taken as an case to verify this work. The result shows that the registration method proposed in this paper is more accurate and stable compared with that based on Vuforia and the augmented assembly guidance system through this method greatly improves the user's time compared with the traditional assembly.

Conclusions

Therefore, we can conclude that the object registration method mentioned in this paper can be well applied in the augmented assembly guidance system, which can do enhance the efficiency of assembly considerably.

Keyword

Registration ; Augmented assembly guidance ; Lucas-Kanade ; Iterative Closet Point ; Complex product

Cite this article

Kangkang YANG, Yu GUO, Pengzhou TANG, Haopeng ZHANG, Han LI. Object registration using an RGB-D camera for complex product augmented assembly guidance. Virtual Reality & Intelligent Hardware, 2020, 2(6): 501-517 DOI:10.1016/j.vrih.2020.01.004

References

1. Nee A Y C, Ong S K, Chryssolouris G, Mourtzis D. Augmented reality applications in design and manufacturing. CIRP Annals, 2012, 61(2): 657–679 DOI:10.1016/j.cirp.2012.05.010

2. Liu J, Wang J. Collaborative assembly design and planning of complex products. Huazhong University of science and technology press, 2011, 56–58

3. Yang J. Research on virtual assembly technologies of aircraft components. Nanjing: Nanjing University of Aeronautics and Astronautics, 2007, 1–4

4. Xiao H. Research on the key technologies of mobile three-dimensional model for assembly site of complex product. Xi'an: Northwestern Polytechnical University, 2014, 1–9

5. Yang H J, Ning R X, Zheng Y. Data exchange from CAD to virtual assembly system based on XML. Machinery Design Manufacture, 2006(11): 46–48 (in Chinese)

6. Liu C, Cao S, Tse W, Xu X. Augmented reality-assisted intelligent window for cyber-physical machine tools. Journal of Manufacturing Systems, 2017, 44: 280–286 DOI:10.1016/j.jmsy.2017.04.008

7. Wu C L, Wang H. A multi-modal augmented reality based virtual assembly system. In: Proceedings of the International Conference on Human-centric Computing 2011 and Embedded and Multimedia Computing 2011. Dordrecht: Springer Netherlands, 2011, 65–72 DOI:10.1007/978-94-007-2105-0_9

8. Fiorentino M, Uva A E, Gattullo M, Debernardis S, Monno G. Augmented reality on large screen for interactive maintenance instructions. Computers in Industry, 2014, 65(2): 270–278 DOI:10.1016/j.compind.2013.11.004

9. Koch C, Neges M, König M, Abramovici M. Natural markers for augmented reality-based indoor navigation and facility maintenance. Automation in Construction, 2014, 48: 18–30 DOI:10.1016/j.autcon.2014.08.009

10. Radkowski R, Oliver J. Natural feature tracking augmented reality for on-site assembly assistance systems//Virtual, Augmented and Mixed Reality. Systems and Applications. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, 281–290 DOI:10.1007/978-3-642-39420-1_30

11. Pasman W, van der Schaaf A, Lagendijk R, Jansen F. Accurate overlaying for mobile augmented reality. Computers Graphics, 1999, 23(6): 875–881 DOI:10.1016/s0097-8493(99)00118-1

12. Järemo Lawin F, Forssén P E, Ovrén H. Efficient multi-frequency phase unwrapping using kernel density estimation// Computer Vision–ECCV 2016. Cham: Springer International Publishing, 2016, 170–185 DOI:10.1007/978-3-319-46493-0_11

13. Zhang Z Y. Flexible camera calibration by viewing a plane from unknown orientations. In: Proceedings of the Seventh IEEE International Conference on Computer Vision. Kerkyra, Greece. IEEE, 1999, 666–673 DOI:10.1109/iccv.1999.791289

14. Garon M, Boulet P O, Doironz J P, Beaulieu L, Lalonde J F. Real-time high resolution 3D data on the HoloLens. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). Merida, Yucatan, Mexico, IEEE, 2016, 189–191 DOI:10.1109/ismar-adjunct.2016.0073

15. Marchand E, Uchiyama H, Spindler F. Pose estimation for augmented reality: a hands-on survey. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(12): 2633–2651 DOI:10.1109/tvcg.2015.2513408

16. Belghit H, Bellarbi A, Zenati N, Otmane S. Vision-based pose estimation for augmented reality: a comparison study. 2018

17. Baker S, Matthews I. Lucas-kanade 20 years on: a unifying framework. International Journal of Computer Vision, 2004, 56(3): 221–255 DOI:10.1023/b:visi.0000011205.11775.fd

18. Besl P J, McKay N D. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 239–256 DOI:10.1109/34.121791

19. Wang Y T, Chen J. Introduction to augmented reality technology. Beijing: The science publishing company, 2015

20. Rosten E, Drummond T. Machine learning for high-speed corner detection//Computer Vision–ECCV 2006. Berlin, Heidelberg, Springer Berlin Heidelberg, 2006, 430–443 DOI:10.1007/11744023_34

21. Arun K S, Huang T S, Blostein S D. Least-squares fitting of two 3-D point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(5): 698–700 DOI:10.1109/tpami.1987.4767965

22. Yang K K, Guo Y, Zhang H P, Tang P Z. Hololens-based multi-channel interactive augmented assembly guidance system for complex products. 2018 International Academic Conference for Graduates, NUAA, 2018

1. Zhiyuan ZHANG, Yuchao DAI, Jiadai SUN, Deep learning based point cloud registration: an overview Virtual Reality & Intelligent Hardware 2020, 2(3): 222-246

2. Wei LYU, Zhong ZHOU, Lang CHEN, Yi ZHOU, A survey on image and video stitching Virtual Reality & Intelligent Hardware 2019, 1(1): 55-83

Full Text

11 PDF

730 HTML

EXPORT

EndNote ProCite Reference Manager BibTex RefWorks

SHARE ON

ARTICLES BY AUTHORS

VRIH

BAIDU SCHOLAR

WANFANG DATA

CNKI