A review on image-based rendering

Abstract

Image-based rendering is important both in the field of computer graphics and computer vision, and it is also widely used in virtual reality technology. For more than two decades, people have done a lot of work on the research of image-based rendering, and these methods can be divided into two categories according to whether the geometric information of the scene is utilized. According to this classification, we introduce some classical methods and representative methods proposed in recent years. We also compare and analyze the basic principles, advantages and disadvantages of different methods. Finally, some suggestions are given for research directions on image-based rendering techniques in the future.

Keyword

Image-based rendering ; Virtual reality ; Image interpolation ; Panorama

Cite this article

Yuan CHANG, Guo-Ping WANG. A review on image-based rendering. Virtual Reality & Intelligent Hardware, 2019, 1(1): 39-54 DOI:10.3724/SP.J.2096-5796.2018.0004

References

1. Snavely N, Seitz S M, Szeliski R. Photo Tourism: Exploring image collections in 3D. ACM Transactions on Graphics, 2006, 25: 835–846 DOI:10.1145/1141911.1141964

2. Furukawa Y, J. Accurate Ponce, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2010, 32(8): 1362–1376 DOI:10.1109/TPAMI.2009.161

3. Google Street View. https: //www. google. com/streetview/

4. Shum H Y, Kang S B. A review of image-based rendering techniques. In: NganK N, SikoraT, SunM T, eds. Proceedings of IEEE/SPIE Visual Communications and Image Processing (VCIP), Perth. Australia, 2000, 2–13

5. Adelson E H, Bergen J R. The plenoptic function and the elements of early vision. In: Landy M S, Movshon J A, eds. Computational Models of Visual Processing. Cambridge: MIT Press, 1991

6. Wong T T, Heng P A, Or S H, Ng W Y. Image-based rendering with controllable illumination. In: Dorsey J, Slusallek P, eds. Proceedings of the 8-th Eurographics Workshop on Rendering. Berlin: Springer-Verlag, 1997, 13–22

7. McMillan L, Bishop G. Plenoptic modeling: An image-based rendering system. In: Mair SG, Cook R, eds. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. New York: ACM, 1995, 39–46

8. Levoy M, Hanrahan P. Light field rendering. In: Fujii J, eds. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. New York: ACM, 1996, 31–42 DOI:10.1145/237170.237199

9. Shum H Y, He L W. Rendering with concentric mosaics. In: Waggenspack W, eds. Proceedings of the 26th annual conference on Computer graphics and interactive techniques. New York: ACM Press/Addison-Wesley Publishing Co., 1999, 299–306 DOI:10.1145/311535.311573

10. Szeliski R, Shum H Y. Creating full view panoramic image mosaics and texture-mapped models. In: OwenG S, WhittedT, Mones-HattalB, eds. Proceedings of the 24th annual conference on Computer graphics and interactive techniques. New York: ACM Press/Addison-Wesley Publishing Co., New York, 1997, 251–258 DOI:10.1145/258734.258861

11. Chen S E. Quick Time VR–an image-based approach to virtual environment navigation. In: Mair SG, Cook R, eds. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. New York: ACM, 1995, 29–38 DOI:10.1145/218380.218395

12. Roundshot. http: //www. roundshot. com

13. Shum H Y, Szeliski R. Construction and refinement of panoramic mosaics with global and local alignment. In: Sixth International Conference on Computer Vision (ICCV’98). Bombay, 1998, 953–958 DOI:10.1109/ICCV.1998.710831

14. Gortler S J, Grzeszczuk R, Szeliski R, Cohen M F. The lumigraph. In: Fujii J, eds. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. New York: ACM, 1996, 43–54 DOI:10.1145/237170.237200

15. Debevec P, Downing G, Bolas M, Peng H-Y, Urbach J. Spherical light field environment capture for virtual reality using a motorized pan/tilt head and offset camera. In: ACM Siggraph 2015 Posters. Los Angeles, California, 2015, 1–1 DOI:10.1145/2787626.2787648

16. Buehler C, Bosse M, McMillan L, Gortler S, Cohen M. Unstructured lumigraph rendering. In: Pocock L, eds. Proceedings of the 28th annual conference on Computer graphics and interactive techniques. New York: ACM, 2001, 425–432 DOI:10.1145/383259.383309

17. Rademacher P. View-dependent geometry. In: Waggenspack W, eds. Proceedings of the 26th annual conference on Computer graphics and interactive techniques. New York: ACM Press/Addison-Wesley Publishing Co., 1999, 439–446 DOI:10.1145/311535.311612

18. Vedula S, Baker S, Kanade T. Spatio-temporal view interpolation. In: Gibson S, Debevec P, eds. Proceedings of the 13th Eurographics Workshop on Rendering. Aire-la-Ville: Eurographics Association, 2002

19. Chaurasia G, Duchene S, Sorkine-Hornung O, Drettakis G. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics, 2013, 32(3): 1–12 DOI:10.1145/2487228.2487238

20. Lipski C, Klose F, Magnor M. Correspondence and depth-image based rendering a hybrid approach for free-viewpoint video. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6): 942–951 DOI:10.1109/TCSVT.2014.2302379

21. Shade J, Gortler S, He L W, Szeliski R. Layered depth images. In: CunninghamS, BransfordW, CohenM F, eds. Proceedings of the 25th annual conference on Computer graphics and interactive techniques. New York: ACM, 1998, 231–242. DOI:10.1145/280814.280882

22. Chang C, Bishop G, Lastra A. LDI tree: A hierarchical representation for image-based rendering. In: Waggenspack W, eds. Proceedings of the 26th annual conference on Computer graphics and interactive techniques. New York: ACM Press/Addison-Wesley Publishing Co., 1999: 291–298 DOI:10.1145/311535.311571

23. Penner E, Zhang L. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics, 2017, 36(6): 1–11 DOI:10.1145/3130800.3130855

24. Chen S, Williams L. View interpolation for image synthesis. In: Whitton MC, eds. Proceedings of the 20th annual conference on Computer graphics and interactive techniques. New York: ACM, 1993, 279–288 DOI:10.1145/166117.166153

25. Seitz S M, Dyer C M. View morphing. In: Fujii J, eds. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. New York: ACM, 1996, 21–30 DOI:10.1145/237170.237196

26. Nie Y W, Zhang Z S, Sun H Q, Su T, Li G. Homography propagation and optimization for wide-baseline street image interpolation. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(10): 2328–2341 DOI:10.1109/TVCG.2016.2618878

27. Kopf J, Langguth F, Scharstein D, Szeliski R, Goesele M. Image-based rendering in the gradient domain. ACM Transactions on Graphics, 2013, 32(6): 1–9 DOI:10.1145/2508363.2508369

28. Sinha S N, Kopf J, Goesele M, Scharstein D. Image-based rendering for scenes with reflections. ACM Transactions on Graphics, 2012, 31(4): 1–10

29. Thonat T, Djelouah A, Durand F, Drettakis G. Thin structures in image based rendering. Computer Graphics Forum, 2018, 37: 107–118

30. Yang H S, Lin W Y, Lu J B. DAISY Filter Flow: A generalized discrete approach to dense correspondences. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, 2014, 3406–3413

31. Bao L C, Yang Q X, Jin H L. Fast edge-preserving PatchMatch for large displacement optical flow. IEEE Transactions on Image Processing, 2014, 23: 4996–5006 DOI:10.1109/CVPR.2014.452

32. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2274–2282 DOI:10.1109/TPAMI.2012.120

33. Fischler M A, Bolles R C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

34. Hawe S, Kleinsteuber M, Diepold K. Dense disparity maps from sparse disparity measurements. In: 2011 International Conference on Computer Vision. Barcelona, 2011, 2126–2133 DOI:10.1109/ICCV.2011.6126488

35. Sinha S N, Steedly D, Szeliski R. Piecewise planar stereo for image-based rendering. In: 2009 IEEE 12th International Conference on Computer Vision. Kyoto, 2009, 1881–1888

36. Gallup D, Frahm J, Pollefeys M. Piecewise planar and non-planar stereo for urban scene reconstruction. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA, 2010, 1418–1425

37. Furukawa Y, Curless B, Seitz S M, Szeliski R. Manhattan-world stereo. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, 2009, 1422–1429 DOI:10.1109/CVPR.2009.5206867

38. Lipski C, Linz C, Neumann T, Wacker M, Magnor M. High resolution image correspondences for video post-produc-tion. In: 2010 Conference on Visual Media Production. London, 2010, 33–39 DOI:33-39.10.1109/CVMP.2010.12

39. Liu C, Yuen J, Torralba A. SIFT flow: Dense correspondence across different scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 978–994 DOI:10.1109/TPAMI.2010.147

40. Hosni A, Bleyer M, Rhemann C, Gelautz M, Rother C. Real-time local stereo matching using guided image filtering. In: 2011 IEEE International Conference on Multimedia and Expo. Barcelona, 2011, 1–6 DOI:10.1109/ICME.2011.6012131

41. Ma Z Y, He K M, Wei Y C. Constant time weighted median filtering for stereo matching and beyond. In: 2013 IEEE International Conference on Computer Vision. Sydney, NSW, 2013, 49–56

42. He K M, Sun J, Tang X O. Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013, 35: 1397–1409 DOI:10.1109/TPAMI.2012.213

43. Hedman P, Alsisan S, Szeliski R, Kopf J. Casual 3D photography. ACM Transactions on Graphics, 2017, 36: 234: 1–15

44. Schönberger J L, Zheng E L, Frahm J M. Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision. Amsterdam, the Netherlands, 2016

45. Hedman P, Kopf J. Instant 3D photography. ACM Transactions on Graphics, 2017, 36(6): 1–15 DOI:10.1145/3130800.3130828

46. Hedman P, Ritschel T, Drettakis G, Brostow G. Scalable inside-out image-based rendering. ACM Transactions on Graphics, 2016, 35(6): 1–11. DOI:10.1145/2980179.2982420

47. Pérez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22: 313–318 DOI:10.1145/1201775.882269

48. Burt P J, Adelson E H. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics, 1983, 2: 217–236 DOI:10.1145/245.247

49. Kalantari N K, Wang T C, Ramamoorthi R. Learning based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 1–10 DOI:10.1145/2980179.2980251

50. Flynn J, Neulander I, Philbin J, Snavely N. DeepStereo: Learning to predict new views from the world’s imagery. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, 2016, 35(6): 1–10 DOI:10.1145/2980179.2980251

1. Yang LI, Dong WU, Jin HUANG, Feng TIAN, Hong'an WANG, Guozhong DAI, Influence of multi-modality on moving target selection in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(3): 303-315

2. Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI, Gesture interaction in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 84-112

3. Athirah SYAMIMI, Yiwei GONG, Ryan LIEW, VR industrial applications―A singapore perspective Virtual Reality & Intelligent Hardware 2020, 2(5): 409-420

4. Jie GUO, Dongdong WENG, Yue LIU, Qiyong CHEN, Yongtian WANG, Analysis of teenagers' preferences and concerns regarding HMDs in education Virtual Reality & Intelligent Hardware 2021, 3(5): 369-382

5. Susu HUANG, Daqing QI, Jiabin YUAN, Huawei TU, Review of studies on target acquisition in virtual reality based on the crossing paradigm Virtual Reality & Intelligent Hardware 2019, 1(3): 251-264

6. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

7. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

8. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

9. Xu PENG, Zhenyu GAO, Yitong DING, Dongfeng ZHAO, Xiaoyu CHI, Study of ghost image suppression in polarized catadioptric virtual reality optical systems Virtual Reality & Intelligent Hardware 2020, 2(1): 70-78

10. Zhiming HU, Sheng LI, Meng GAI, Temporal continuity of visual attention for future gaze prediction in immersive virtual reality Virtual Reality & Intelligent Hardware 2020, 2(2): 142-152

11. Lihui HUANG, Siti Faatihah Binte Mohd TAIB, Ryan Aung BA, Zhe An GOH, Mengshan XU, Virtual reality research and development in NTU Virtual Reality & Intelligent Hardware 2020, 2(5): 394-408

12. Stéphanie PHILIPPE, Alexis D. SOUCHET, Petros LAMERAS, Panagiotis PETRIDIS, Julien CAPORAL, Gildas COLDEBOEUF, Hadrien DUZAN, Multimodal teaching, learning and training in virtual reality: a review and case study Virtual Reality & Intelligent Hardware 2020, 2(5): 421-442

13. Jia Ming LEE, Xinxing XIA, Clemen OW, Felix CHUA, Yunqing GUAN, VEGO: A novel design towards customizable and adjustable head-mounted display for VR Virtual Reality & Intelligent Hardware 2020, 2(5): 443-453

14. Jingcheng QIAN, Yancong MA, Zhigeng PAN, Xubo YANG, Effects of Virtual-real fusion on immersion, presence, and learning performance in laboratory education Virtual Reality & Intelligent Hardware 2020, 2(6): 569-584

15. Hengwei XU, Siru LI, Wenpeng SONG, Jiajun SUN, Xinli WU, Xiaoqi WANG, Wenzhen YANG, Zhigeng PAN, Abdennour EI RHALIBI, Thermal perception method of virtual chemistry experiments Virtual Reality & Intelligent Hardware 2020, 2(4): 305-315

16. TJ MATTHEWS, Feng TIAN, Tom DOLBY, Interaction design for paediatric emergency VR training Virtual Reality & Intelligent Hardware 2020, 2(4): 330-344

17. Hongxin ZHANG, Jin ZHANG, Xue YIN, Kan ZHOU, Zhigeng PAN, Abdennour EI RHALIBI, Cloud-to-end rendering and storage management for virtual reality in experimental education Virtual Reality & Intelligent Hardware 2020, 2(4): 368-380

18. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

19. Haoyu WANG, Jianhuang WU, A virtual reality based surgical skills training simulator for catheter ablation with real-time and robust interaction Virtual Reality & Intelligent Hardware 2021, 3(4): 302-314

20. Na ZHANG, Liwen TAN, Fengying LI, Bing HAN, Yifa XU, Development and application of digital assistive teaching system for anatomy Virtual Reality & Intelligent Hardware 2021, 3(4): 315-335

21. Daniel VANKOV, David JANKOVSZKY, Effects of using headset-delivered virtual reality in road safety research: A systematic review of empirical studies Virtual Reality & Intelligent Hardware 2021, 3(5): 351-368

22. Dangxiao WANG, Yuan GUO, Shiyi LIU, Yuru ZHANG, Weiliang XU, Jing XIAO, Haptic display for virtual reality: progress and challenges Virtual Reality & Intelligent Hardware 2019, 1(2): 136-162

23. Aiguo SONG, Liyue FU, Multi-dimensional force sensor for haptic interaction: a review Virtual Reality & Intelligent Hardware 2019, 1(2): 121-135

24. Wenmin ZHU, Xiumin FAN, Yanxin ZHANG, Applications and research trends of digital human models in the manufacturing industry Virtual Reality & Intelligent Hardware 2019, 1(6): 558-579

25. Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN, Affine transformation of virtual 3D object using 2D localization of fingertips Virtual Reality & Intelligent Hardware 2020, 2(6): 534-555

26. Yuan WEI, Dongdong GUAN, Qiuchen WANG, Xiangxian LI, Yulong BIAN, Pu QIN, Yanning XU, Chenglei YANG, Virtual fire drill system supporting co-located collaboration Virtual Reality & Intelligent Hardware 2019, 1(3): 290-302