A survey on image and video stitching

Abstract

Image/video stitching is a technology for solving the field of view (FOV) limitation of images/videos. It stitches multiple overlapping images/videos to generate a wide-FOV image/video, and has been used in various fields such as sports broadcasting, video surveillance, street view, and entertainment. This survey reviews image/video stitching algorithms, with a particular focus on those developed in recent years. Image stitching first calculates the corresponding relationships between multiple overlapping images, deforms and aligns the matched images, and then blends the aligned images to generate a wide-FOV image. A seamless method is always adopted to eliminate such potential flaws as ghosting and blurring caused by parallax or objects moving across the overlapping regions. Video stitching is the further extension of image stitching. It usually stitches selected frames of original videos to generate a stitching template by performing image stitching algorithms, and the subsequent frames can then be stitched according to the template. Video stitching is more complicated with moving objects or violent camera movement, because these factors introduce jitter, shakiness, ghosting, and blurring. Foreground detection technique is usually combined into stitching to eliminate ghosting and blurring, while video stabilization algorithms are adopted to solve the jitter and shakiness. This paper further discusses panoramic stitching as a special-extension of image/video stitching. Panoramic stitching is currently the most widely used application in stitching. This survey reviews the latest image/video stitching methods, and introduces the fundamental principles/advantages/weaknesses of image/video stitching algorithms. Image/video stitching faces long-term challenges such as wide baseline, large parallax, and low-texture problem in the overlapping region. New technologies may present new opportunities to address these issues, such as deep learning-based semantic correspondence, and 3D image stitching. Finally, this survey discusses the challenges of image/video stitching and proposes potential solutions.

Keyword

Image stitching ; Video stitching ; Panoramic stitching ; Registration ; Alignment ; Mesh optimization ; Deep learning ; 3D stitching

Cite this article

Wei LYU, Zhong ZHOU, Lang CHEN, Yi ZHOU. A survey on image and video stitching. Virtual Reality & Intelligent Hardware, 2019, 1(1): 55-83 DOI:10.3724/SP.J.2096-5796.2018.0008

References

1. Szeliski R. Image Alignment and Stitching: A Tutorial. Foundations and Trends® in Computer Graphics and Vision,2007; 2(1): 1–104 DOI:10.1561/0600000009

2. Kaynig V, Fischer B, Buhmann J M. Probabilistic image registration and anomaly detection by nonlinear warping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, Alaska, USA, 2008, 1–8 DOI:10.1109/CVPR.2008.4587743

3. Silva R M A, Gomes P B, Frensh T, Monteiro D. Real time 360° video stitching and streaming. In: ACM SIGGRAPH 2016 Posters. Anaheim, California: ACM, 2016: 1–2 DOI:10.1145/2945078.2945148

4. Peleg S, Rousso B, Rav-Acha A, Zomet A. Mosaicing on adaptive manifolds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(10): 1144–1154 DOI:10.1109/34.879794

5. Levin A, Zomet A, Peleg S, Weiss Y. Seamless image stitching in the gradient domain. In: Computer Vision-ECCV 2004: 2004// 2004. Berlin, Heidelberg. Springer Berlin Heidelberg, 2004, 377–389

6. Zomet A, Levin A, Peleg S, Weiss Y. Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing, 2006, 15(4): 969–977 DOI:10.1109/TIP.2005.863958

7. Jia J, Tang C K. Eliminating structure and intensity misalignment in image stitching. In: Tenth IEEE International Conference on Computer Vision (ICCV'05). Beijing, China, 2005, 1651–1658 DOI:10.1109/ICCV.2005.87

8. Brown M, Lowe D G. Recognizing Panoramas. In: Proceedings of the IEEE International Conference on Computer Vision. Nice, France, 2003, 3: 1218

9. Brown M, Lowe D G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 2007, 74(1): 59–73 DOI:10.1007/s11263-006-0002-3

10. Gao J, Kim S J, Brown M S. Constructing image panoramas using dual-homography warping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO, USA, 201, 49–56 DOI:10.1109/CVPR.2011.5995433

11. Duffin K L, Barrett W A. Fast focal length solution in partial panoramic image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Kauai, HI, USA, 2001, II–II DOI:10.1109/CVPR.2001.991031

12. Lin W Y, Liu S, Matsushita Y, Ng T T, Cheong L F. Smoothly varying affine stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO, USA, 2011, 345–352 DOI:10.1109/CVPR.2011.5995314

13. Zaragoza J, Chin T J, Brown M S, Suter D. As-projective-as-possible image stitching with moving DLT. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, 2013, 2339–2346 DOI:10.1109/CVPR.2013.303

14. He B, Yu S. Parallax-Robust Surveillance Video Stitching. 2016, 16(1):7 DOI:10.3390/s16010007

15. Su T, Nie Y, Zhang Z, Sun H, Li G. Video stitching for hand-held inputs via combined video stabilization. In: SIGGRAPH ASIA 2016 Technical Briefs. Macao, China, 2016: 25

16. Nie Y, Su T, Zhang Z, Sun H, Li G. Dynamic Video Stitching via Shakiness Removing. IEEE Transactions on Image Processing, 2018, 27(1): 164–178 DOI:10.1109/TIP.2017.2736603

17. Lin K, Liu S, Cheong L F, Zeng B. Seamless video stitching from hand‐held camera inputs. Computer Graphics Forum, 2016, 35(2): 479–487 DOI:10.1111/cgf.12848

18. Matsushita Y, Ofek E, Weina G, Xiaoou T, Heung-Yeung S. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(7):1150–1163 DOI:10.1109/TPAMI.2006.141

19. Liu S, Yuan L, Tan P, Sun J. Bundled camera paths for video stabilization. ACM Trans. Graph. 2013, 32(4): 1–10 DOI:10.1145/2461912.2461995

20. Szeliski R, Shum H Y. Creating full view panoramic image mosaics and texture-mapped models. In: SIGGRAPH. Los Angeles, 1997, 251–258

21. Lee J, Kim B, Kim K, Kim Y, Noh J. Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Transactions on Graphics (TOG), 2016, 35(4): 1–11 DOI:10.1145/2897824.2925983

22. Zhang Z. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2000, 22: 1330–1334

23. Zhi Q, Cooperstock J R. Toward dynamic image mosaic generation with robustness to parallax. IEEE Transactions on Image Processing, 2012, 21(1): 366–378 DOI:10.1109/TIP.2011.2162743

24. Uyttendaele M, Eden A, Skeliski R. Eliminating ghosting and exposure artifacts in image mosaics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Kauai, HI, USA, 2001, II–II DOI:10.1109/CVPR.2001.991005

25. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 DOI:10.1023/B:VISI.0000029664.99615.94

26. Harris C, Stephens M. A combined corner and edge detector. In: Proceedings of Alvey vision conference. Manchester, UK, 1988

27. Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346–359 DOI:10.1016/j.cviu.2007.09.014

28. Rosten E, Drummond T. Machine Learning for High-Speed Corner Detection. In: Computer Vision – ECCV 2006. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, 430–443

29. Liu W X, Chin T J. Correspondence Insertion for As-Projective-As-Possible Image Stitching. 2016

30. Chang C H, Sato Y, Chuang Y Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA, 2014, 3254–3261

31. Lin K, Jiang N, L-F Cheong , Do M, Lu J. Seagull: seam-guided local alignment for parallax-tolerant image stitching. In: Computer Vision – ECCV 2016. Cham: Springer International Publishing, 2016, 370–385

32. Herrmann C, Wang C, Bowen R S, Keyder E, Zabih R. Object-Centered Image Stitching. In: Computer Vision – ECCV 2018. Springer International Publishing, 2018, 846–861 DOI:10.1007/978-3-030-01219-9_50

33. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

34. Guo H, Liu S, He T, Zhu S, Zeng B, Gabbouj M. Joint Video Stitching and Stabilization From Moving Cameras. IEEE Transactions on Image Processing, 2016, 25(11): 5491–5503 DOI:10.1109/TIP.2016.2607419

35. Zou D, Tan P. Coslam: Collaborative visual slam in dynamic environments. IEEE transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 354–366 DOI:10.1109/TPAMI.2012.104

36. Brown M, Hartley R I, Nistér D. Minimal solutions for panoramic stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, Minnesota, USA, 2007, 1–8 DOI:10.1109/CVPR.2007.383082

37. Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Gross M. Panoramic video from unstructured camera arrays. Computer Graphics Forum. 2015, 34(2): 57–68 DOI:10.1111/cgf.12541

38. Flynn J, Neulander I, Philbin J, Snavely N. Deepstereo: Learning to predict new views from the world's imagery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Nevada, 2016, 5515–5524

39. Ufer N, Ommer B. Deep semantic feature matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA, 2017, 5929–5938 DOI:10.1109/CVPR.2017.628

40. Deng J, Dong W, Socher R, Li L, Kai L, Li F-F. ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Miami, Florida, USA 2009, 248–255 DOI:10.1109/CVPR.2009.5206848

41. Aberman K, Liao J, Shi M, et al. Neural Best-Buddies: Sparse Cross-Domain Correspondence. ACM Transactions on Graphics (TOG), 2018, 37(4): 69 DOI:10.1145/3197517.3201332

42. Rocco I, Arandjelovic R, Sivic J. Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA, 2017, 39–48 DOI:10.1109/CVPR.2017.12

43. Jeon S, Kim S, Min D, Sohn K. PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondence Estimation. In: Proceedings of European Conference on Computer Vision. Munich, Germany, 2018, 355–371

44. DOI: 10.1007/978-3-030-01231-1_22Zhou Y, Cao M, You J, Meng M, Wang Y, Zhou Z. MR Video Fusion: Interactive 3D Modeling and Stitching on Wide-baseline Videos. In: ACM Symposium on Virtual Reality Software and Technology. Tokyo, Japan: ACM, 2018, 1–11 DOI:10.1145/3281505.3281513

45. Bujnák M, Sara R. A robust graph-based method for the general correspondence problem demonstrated on image stitching. In:2007 IEEE 11th International Conference on Computer Vision. Rio de Janeiro, Brazil, 2007, 1–8 DOI:10.1109/ICCV.2007.4408884

46. Šára R. The principle of stability applied to matching problems in computer vision. RR CTU–CMP–2007–17. Center for Machine Perception, Czech Technical University, 2007

47. Collins R T. A space-sweep approach to true multi-image matching. In: Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA, USA, 1996, 358–363 DOI:10.1109/CVPR.1996.517097

48. Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(9): 1124–1137 DOI:10.1109/TPAMI.2004.60

49. Lin C C, Pankanti S U, Natesan Ramamurthy K, Aravkin A Y. Adaptive as-natural-as-possible image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA, 2015,1155–1163

50. Li N, Xu Y, Wang C. Quasi-homography warps in image stitching. IEEE Transactions on Multimedia, 2018, 20(6): 1365–1375 DOI:10.1109/TMM.2017.2771566

51. Zhang F, Liu F. Parallax-tolerant image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA, 2014, 3262–3269

52. Liu F, Gleicher M, Jin H, Agarwala A. Content-preserving warps for 3D video stabilization. In: Acm Siggraph. 2009, 1–9 DOI:10.1145/1576246.1531350

53. Gao J, Li Y, Chin T J, Brown M S. Seam-Driven Image Stitching. In: Proceedings of Euro-Graphics. Girona, Spain, 2013, 45–48

54. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In: International Conference on Neural Information Processing Systems, 2015, 91–99

55. Herrmann C, Wang C, Bowen R S, Keyder E, Krainin M, Liu C, Zabih R. Robust Image Stitching with Multiple Registrations. In: Computer Vision – ECCV 2018. Munich, Germany: Cham: Springer International Publishing, 2018, 53–69 DOI:10.1007/978-3-030-01216-8_4

56. Chen Y S, Chuang Y Y. Natural Image Stitching with the Global Similarity Prior. In: Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016, 186–201 DOI:10.1007/978-3-319-46454-1_12

57. Zhang G, He Y, Chen W, Jia J, Bao H. Multi-Viewpoint Panorama Construction With Wide-Baseline Images. IEEE Transactions on Image Processing, 2016, 25(7): 3099–3111 DOI:10.1109/TIP.2016.2535225

58. Xiang T Z, Xia G S, Bai X, Zhang L. Image stitching by line-guided local warping with global similarity constraint. Pattern Recognition, 2018, 83:481–497 DOI:10.1016/j.patcog.2018.06.013

59. Barnes C, Shechtman E, Finkelstein A, Goldman D B. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 2009, 28(3): 1–11 DOI:10.1145/1531326.1531330

60. Rav-Acha A, Pritch Y, Lischinski D, Peleg S. Dynamosaics: video mosaics with non-chronological time. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, CA, USA, 2005, 58–65 DOI:10.1109/CVPR.2005.137

61. Jiang W, Gu J. Video stitching with spatial-temporal content-preserving warping. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, 42–48 DOI:10.1109/CVPRW.2015.7301374

62. Kwatra V, Schödl A, Essa I A, Turk G, Bobick A F. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (ToG), 2003, 22(3): 277–286 DOI:10.1145/1201775.882264

63. Shum H Y, Szeliski R. Construction and refinement of panoramic mosaics with global and local alignment. In: Proceedings of the IEEE International Conference on Computer Vision. Bombay, India, 1998, 953–956 DOI:10.1109/ICCV.1998.710831

64. Steedly D, Pal C, Szeliski R. Efficiently registering video into panoramic mosaics. In: Proceedings of the IEEE International Conference on Computer Vision. Beijing, China, 2005, 2: 1300–1307 DOI:10.1109/ICCV.2005.86

65. He K, Chang H, Sun J. Rectangling panoramic images via warping. ACM Transactions on Graphics (TOG), 2013, 32(4): 1–10 DOI:10.1145/2461912.2462004

66. Avidan S, Shamir A. Seam carving for content-aware image resizing. ACM Transactions on Graphics (TOG), 2007, 26(3): 10 DOI:10.1145/1276377.1276390

67. Galil Z. Efficient algorithms for finding maximal matching in graphs. In: Colloquium on Trees in Algebra and Programming. Berlin, Heidelberg: Springer Berlin Heidelberg, 1983, 90–113

68. Pan J, Appia V, Villarreal J, Weaver L, Kwon D. Rear-Stitched View Panorama: A Low-Power Embedded Implementation for Smart Rear-View Mirrors on Vehicles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI, USA, 2017, 1184–1193 DOI:10.1109/CVPRW.2017.157

69. Agarwala A, Agrawala M, Cohen M, Salesin D, Szeliski R. Photographing long scenes with multi-viewpoint panoramas. ACM Transactions on Graphics (TOG), 2006, 25(3): 853–861 DOI:10.1145/1141911.1141966

70. Brown M, Lowe D G. Unsupervised 3D object recognition and reconstruction in unordered datasets. In: Proceedings of the International Conference on 3-D Digital Imaging and Modeling. Ottawa, Ontario, Canada, 2005, 56–63 DOI:10.1109/3DIM.2005.81

71. Ho T, Budagavi M. Dual-fisheye lens stitching for 360-degree imaging. In: Proceedings of the International Conference on Acoustics, Speech and Signal Processing. New Orleans, USA, 2017, 2172–2176 DOI:10.1109/ICASSP.2017.7952541

72. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In: Proceedings of Conference on Neural Information Processing Systems. Lake Tahoe, Nevada, USA, 2012, 1097–1105

2. Lianyu ZHENG, Xinyu LIU, Zewu AN, Shufei LI, Renjie ZHANG, A smart assistance system for cable assembly by combining wearable augmented reality with portable visual inspection Virtual Reality & Intelligent Hardware 2020, 2(1): 12-27

3. Muhammad IRFAN, Muhammad MUNSIF, Deepdive: a learning-based approach for virtual camera in immersive contents Virtual Reality & Intelligent Hardware 2022, 4(3): 247-262

4. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460

5. Yuanyuan SHI, Yunan LI, Xiaolong FU, Kaibin MIAO, Qiguang MIAO, Review of dynamic gesture recognition Virtual Reality & Intelligent Hardware 2021, 3(3): 183-206

6. Xiaojiao SONG, Jianjun ZHU, Jingfan FAN, Danni AI, Jian YANG, Topological distance-constrained feature descriptor learning model for vessel matching in coronary angiographies Virtual Reality & Intelligent Hardware 2021, 3(4): 287-301

7. Rafik HAMZA, Minh-Son DAO, Privacy-preserving deep learning techniques for wearable sensor-based big data applications Virtual Reality & Intelligent Hardware 2022, 4(3): 210-222

8. Kangkang YANG, Yu GUO, Pengzhou TANG, Haopeng ZHANG, Han LI, Object registration using an RGB-D camera for complex product augmented assembly guidance Virtual Reality & Intelligent Hardware 2020, 2(6): 501-517

9. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246