|
|
|
Joint view synthesis and disparity refinement for stereo matching |
Gaochang WU1,2, Yipeng LI2, Yuanhao HUANG3, Yebin LIU2( ) |
1. State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang 110819, China
2. Broadband Network & Digital Media Lab, Department of Automation, Tsinghua University, Beijing 100084, China
3. Orbbec Company, Shenzhen 518061, China |
|
|
|
|
Abstract Typical stereo algorithms treat disparity estimation and view synthesis as two sequential procedures. In this paper, we consider stereo matching and view synthesis as two complementary components, and present a novel iterative refinement model for joint view synthesis and disparity refinement. To achieve the mutual promotion between view synthesis and disparity refinement, we apply two key strategies, disparity maps fusion and disparity-assisted plane sweep-based rendering (DAPSR). On the one hand, the disparity maps fusion strategy is applied to generate disparity map from synthesized view and input views. This strategy is able to detect and counteract disparity errors caused by potential artifacts from synthesized view. On the other hand, the DAPSR is used for view synthesis and updating, and is able to weaken the interpolation errors caused by outliers in the disparity maps. Experiments onMiddlebury benchmarks demonstrate that by introducing the synthesized view, disparity errors due to large occluded region and large baseline are eliminated effectively and the synthesis quality is greatly improved.
|
| Keywords
stereo matching
view synthesis
disparity refinement
|
|
Corresponding Author(s):
Yebin LIU
|
|
Just Accepted Date: 08 August 2018
Online First Date: 26 February 2019
Issue Date: 19 July 2019
|
|
| 1 |
M Bleyer, C Rother, P Kohli. Surface stereo with soft segmentation. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1570–1577
|
| 2 |
G P Fickel, C R Jung, T Malzbender, R Samadani, B Culbertson. Stereo matching and view interpolation based on image domain triangulation. IEEE Transactions on Image Processing, 2013, 22(9): 3353–3365
https://doi.org/10.1109/TIP.2013.2264819
|
| 3 |
C Vogel, K Schindler, S Roth. 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision, 2015, 115(1): 1–28
https://doi.org/10.1007/s11263-015-0806-0
|
| 4 |
G Wu, B Masia, A Jarabo, Y Zhang, L Wang, Q Dai, T Chai, Y Liu. Light field image processing: an overview. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(7): 926–954
https://doi.org/10.1109/JSTSP.2017.2747126
|
| 5 |
G Wu, Y Liu, L Fang, Q Dai, T Chai. Light field reconstruction using deep convolutional network on EPI and extended applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(7): 1681–1694
https://doi.org/10.1109/TPAML.2018.2845393
|
| 6 |
C Fehn. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems XI. 2004, 93–104
https://doi.org/10.1117/12.524762
|
| 7 |
A Hosni, C Rhemann, M Bleyer, C Rother, M Gelautz. Fast costvolume filtering for visual correspondence and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 504–511
https://doi.org/10.1109/TPAMI.2012.156
|
| 8 |
S Lee, J H Lee, J Lim, I H Suh. Robust stereo matching using adaptive random walk with restart algorithm. Image and Vision Computing, 2015, 37(C): 1–11
https://doi.org/10.1016/j.imavis.2015.01.003
|
| 9 |
D Sun, S Roth, M J Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, 2014, 106(2): 115–137
https://doi.org/10.1007/s11263-013-0644-x
|
| 10 |
A Kubota, A Smolic, M Magnor, M Tanimoto, T Chen, C Zhang. Multiview imaging and 3DTV. IEEE Signal Processing Magazine, 2007, 24(6): 10–21
https://doi.org/10.1109/MSP.2007.905873
|
| 11 |
N Stefanoski, O Wang, M Lang, P Greisen, S Heinzle, A Smolic. Automatic view synthesis by image-domain-warping. IEEE Transactions on Image Processing, 2013, 22(9): 3329–3341
https://doi.org/10.1109/TIP.2013.2264817
|
| 12 |
D Tian, P L Lai, P Lopez, C Gomila. View synthesis techniques for 3D video. In: Proceedings of Internatlbnal Society for Optics and Phontonics, Applications of Digital Image Processing XXXII. 2009, 7443: 74430T
|
| 13 |
Y Mori, N Fukushima, T Yendo, T Fujii, M Tanimoto. View generation with 3D warping using depth information for FTV. Signal Processing: Image Communication, 2009, 24(1–2): 65–72
https://doi.org/10.1016/j.image.2008.10.013
|
| 14 |
S Wanner, B Goldluecke. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 606–619
https://doi.org/10.1109/TPAMI.2013.147
|
| 15 |
L Zhang, Y H Zhang, H Huang. Efficient variational light field view synthesis for making stereoscopic 3D images. Computer Graphics Forum. 2015, 7(34): 183–191
https://doi.org/10.1111/cgf.12757
|
| 16 |
C Zhang, Z Li, Y Cheng, R Cai, H Chao, Y Rui. Meshstereo: a global stereo model with mesh alignment regularization for view interpolation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2057–2065
https://doi.org/10.1109/ICCV.2015.238
|
| 17 |
N K Kalantari, T C Wang, R Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 193
https://doi.org/10.1145/2980179.2980251
|
| 18 |
D Scharstein, R Szeliski. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. International Journal of Computer Vision, 2002, 47(1–3): 7–42
https://doi.org/10.1023/A:1014573219977
|
| 19 |
G A Kordelas, D S Alexiadis, P Daras, E Izquierdo. Enhanced disparity estimation in stereo images. Image and Vision Computing, 2015, 35: 31–49
https://doi.org/10.1016/j.imavis.2014.12.001
|
| 20 |
Y Peng, G Li, R Wang, W Wang. Stereo matching with spaceconstrained cost aggregation and segmentation-based disparity refinement. In: Proceedings of International Society for Optics and Phoritonics, Three-Dimensional Image Processing, Measurement, and Applications. 2015, 9393: 939309
|
| 21 |
J Zbontar, Y LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 2016, 17(1): 2287–2318
|
| 22 |
W Luo, A G Schwing, R Urtasun. Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 5695–5703
https://doi.org/10.1109/CVPR.2016.614
|
| 23 |
S Mattoccia, M Viti, F Ries. Near real-time fast bilateral stereo on the GPU. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2011, 136–143
https://doi.org/10.1109/CVPRW.2011.5981835
|
| 24 |
S Mattoccia, S Giardino, A Gambini. Accurate and efficient cost aggregation strategy for stereo correspondence based on approximated joint bilateral filtering. In: Proceedings of Asian Conference on Computer Vision. 2009, 371–380
|
| 25 |
K J Yoon, I S Kweon. Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(4): 650–656
https://doi.org/10.1109/TPAMI.2006.70
|
| 26 |
V Kolmogorov, R Zabih. Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 508–515
https://doi.org/10.1109/ICCV.2001.937668
|
| 27 |
T Taniai, Y Matsushita, T Naemura. Graph cut based continuous stereo matching using locally shared labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1613–1620
https://doi.org/10.1109/CVPR.2014.209
|
| 28 |
Q Yang, L Wang, R Yang, H Stewénius, D Nistér. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 492–504
https://doi.org/10.1109/TPAMI.2008.99
|
| 29 |
F Guney, A Geiger. Displets: resolving stereo ambiguities using object knowledge. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4165–4175
https://doi.org/10.1109/CVPR.2015.7299044
|
| 30 |
E T Psota, J Kowalczuk, M Mittek, L C Perez. Map disparity estimation using hidden markov trees. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2219–2227
https://doi.org/10.1109/ICCV.2015.256
|
| 31 |
M G Mozerov, W J Van de. Accurate stereo matching by two-step energy minimization. IEEE Transactions on Image Processing, 2015, 24(3): 1153–1163
https://doi.org/10.1109/TIP.2015.2395820
|
| 32 |
X Mei, X Sun, M Zhou, S Jiao, H Wang, X Zhang. On building an accurate stereo matching system on graphics hardware. In: Proceedings of the IEEE Conference on Computer Vision Workshops. 2011, 467–474
https://doi.org/10.1109/ICCVW.2011.6130280
|
| 33 |
O Veksler. Fast variable window for stereo correspondence using integral images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 556–561
https://doi.org/10.1109/CVPR.2003.1211403
|
| 34 |
A Klaus, M Sormann, K Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th International Conference on Pattern Recognition. 2006, 15–18
https://doi.org/10.1109/ICPR.2006.1033
|
| 35 |
C Vogel, K Schindler, S Roth. 3D scene flow estimation with a rigid motion prior. In: Proceedings of the IEEE International Conference on Computer Vision. 2011, 1291–1298
https://doi.org/10.1109/ICCV.2011.6126381
|
| 36 |
Y Boykov, O Veksler, R Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11): 1222–1239
https://doi.org/10.1109/34.969114
|
| 37 |
Q Yang, R Yang, J Davis, D Nistér. Spatial-depth super resolution for range images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8
https://doi.org/10.1109/CVPR.2007.383211
|
| 38 |
V Boominathan, K Mitra, A Veeraraghavan. Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In: Proceedings of the IEEE International Conference on Computational Photography. 2014, 1–10
https://doi.org/10.1109/ICCPHOT.2014.6831814
|
| 39 |
Z Ma, K He, Y Wei, J Sun, E Wu. Constant time weighted median filtering for stereo matching and beyond. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 49–56
https://doi.org/10.1109/ICCV.2013.13
|
| 40 |
Q Zhang, L Xu, J Jia. 100+ times faster weighted median filter (WMF). In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2830–2837
https://doi.org/10.1109/CVPR.2014.362
|
| 41 |
Z Lin, H Y Shum. A geometric analysis of light field rendering. International Journal of Computer Vision, 2004, 58(2): 121–138
https://doi.org/10.1023/B:VISI.0000015916.91741.27
|
| 42 |
D Scharstein, R Szeliski. High-accuracy stereo depth maps using structured light. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 195–202
https://doi.org/10.1109/CVPR.2003.1211354
|
| 43 |
D Scharstein, H Hirschmüller, Y Kitajima, G Krathwohl, N Nešić, X Wang, P Westling. High-resolution stereo datasets with subpixelaccurate ground truth. In: Proceedings of German Conference on Pattern Recognition. 2014, 31–42
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
