1. College of Mechanical & Electrical Engineering, Wenzhou University, Wenzhou 325035, China; 2. School of Mechatmnics Engineering and Automation, Shanghai University, Shanghai 200072, China
罗胜, 龚振邦, 马光. 多视重建的研究现状[J]. 机器人, 2010, 32(5): 695-704..
LUO Sheng, GONG Zhenbang, MA Guang. A Review of Today's Multi-view Reconstruction. ROBOT, 2010, 32(5): 695-704..
Abstract：This paper reviews today's algorithms about multi-view reconstruction.Firstly,the classification and evaluation methods for multi-view reconstruction are introduced.Secondly,some constraints,such as photo-consistency and visibility, are analyzed.Then,some typical algorithms are discussed,such as depth map,space carving,deformable model,objective function optimization(including the level set method,graph cuts method),seed-growing method,and probability estimation method(including Markov-field method,EM method).Some shortcomings in all of these algorithms are discussed.A new 3D reconstruction idea integrating some of aforementioned reconstruction methods and reconstruction cues is proposed to improve the integrity and accuracy of multi-view reconstruction.
[1] Broadhurst A.A prohabilistic framework for space carving[D]. Cambridge,UK:Cambridge University,2001.
[2] Kutulakos K N,Seitz S M.A theory of shape by space carving[J]. International Journal of Computer Vision,2000,38(3):199-218.
[3] Yang R,Pollefeys M,Welch G.Dealing with textureless regions and specular highlights:A progressive space carving scheme using a novel photo-consistency measure[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2003:576-584.
[4] Osher S J,Fedkiw R P.Level set methods and dynamic implicit surfaces[M]. Berlin,Germany:Springer-Verlag,2002.
[5] Enright D,Losasso F,Fedkiw R.A fast and accurate semiLagrangian particle level set method[J]. Computers and Structures,2005,83(6/7):479-490.
[6] Faugeras O,Keriven R.Complete dense stereovision using level set methods[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,1998:379-393.
[7] Sung J,Kanade T,Kim D.A unified gradient-based approach for combining ASM into AAM[J]. International Journal of Computer Vision,2007,75(2):297-309.
[8] Solem J E,Overgaard N C.A geometric formulation of gradient descent for variational problems with moving surfaces[M]//Lecture Notes in Computer Science:vol.3459.Berlin,Germany:Springer-Verlag,2005:419-430.
[9] Solem J E,Overgaard N C.A gradient descent procedure for variational dynamic surface problems with constraints[M]//Lecture Notes in Computer Science:vol.3752.Berlin,Germany:Springer-Verlag,2005:332-343.
[10] Memisevic R,Hinton G.Improving dimensionality reduction with spectral gradient descent[J]. Neural Networks,2005,18(5/6):702-710.
[11] Kolmogorov V,Zabih R.Multi-camera scene reconstruction via graph cuts[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2002:82-96.
[12] Paris S,Sillion F X,Quan L.A surface reconstruction method using global graph cut optimization[J]. International Journal of Computer Vision,2006,66(2):141-161.
[13] Vogiatzis G,Torr P H S,Cipolla R.Multi-view stereo via volumetric graph-cuts[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2005:391-398.
[14] Boykov Y,Kolmogorov V.Computing geodesics and minimal surfaces via graph cuts[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2003:26-33.
[15] Zach C.Fast and high quality fusion of depth maps[C]//International Symposium on 3D Data Processing,Visualization and Transmission.2008:321-342.
[16] Campbell N D,Vogiatzis G,Hernandez C,et al.Using multiple hypotheses to improve depth-maps for multi-view stereo[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2008:766-779.
[17] Liu Y B,Cao X,Dai Q H,et al.Continuous depth estimation for multi-view stereo[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2009:2121-2128.
[18] Zitnick C L,Kang S B,Uyttendaele M,et al.High-quality video view interpolation using a layered representation[J]. ACM Transactions on Graphics,2004,23(3):600-608.
[19] Motta A,Damiani C,Del Guerra A,et al.Use of a fast EM algorithm for 3D image reconstruction with the YAP-PET tomograph[J]. Computerized Medical Imaging and Graphics,2002,26(5):293-302.
[20] Mottaa A,Del Guerraa A,Belcaria N,et al.Fast 3D-EM reconstruction using planograms for stationary planar positron emission mammography camera[J]. Computerized Medical Imaging and Graphics,2005,29(8):587-596.
[21] Seitz S,Curless B,James D,et al.A comparison and evaluation of multi-view stereo reconstruction algorithms[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2006:519-526.
[22] Pons J P,Keriven R,Faugeras O,et al.Variational stereovision and 3D scene flow estimation with statistical similarity measures[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2003:597-602.
[23] Eid A,Farag A.On the performance evaluation of 3D reconstruction techniques from a sequence of images[J]. Eurasip Journal on Applied Signal Processing,2005(13):1948-1955.
[24] Hornung A,Kobbelt L.Hierarchical volumetric multi-view stereo reconstruction of manifold surfaces based on dual graph embedding[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2006:503-510.
[25] Auclair A,Vincent N,Cohen L D.Using point correspondences without projective deformation for multi-view stereo reconstruction[C]//IEEE International Conference on Image Processing.Piscataway,NJ,USA:IEEE,2008:193-196.
[26] Bradley D,Boubekeur T,Heidrich F.Accurate multi-view reconstruction using robust binocular stereo and surface meshing[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2008:3498-3505.
[27] Delaunoy A,Prados E,Gargallo P,et al.Minimizing the multiview stereo reprojection error for triangular surface meshes[C]//British Machine Vision Conference.2008:56-72.
[28] Furukawa Y,Ponce J.Accurate,dense,and robust multi-view stereopsis[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2007:1-8.
[29] Jancosek M,Pajdla T.Segmentation based multi-view stereo[C]//Computer Vision Winter Workshop.2009:91-97.
[30] Kolev K,Klodt M,Brox T,et al.Continuous global optimization in multiview 3D reconstruction[J]. International Journal of Computer Vision,2009,84(1):80-96.
[31] Hiep V H,Keriven R,Labatut P,et al.Towards high-resolution large-scale multi-view stereo[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2009:1430-1437.
[32] Goesele M,Snavely N,Cudess B,et al.Multi-view stereo for community photo collections[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2007:741-748.
[33] Starck J,Hilton A.Model-based human shape reconstruction from multiple views[J]. Computer Vision and Image Understanding,2008,111(2):179-194.
[34] Pons J P,Boissonnat J D.Delaunay deformable models:Topology-adaptive meshes based on the restricted Delaunay triangulation[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2007:375-382.
[35] Seitz S M,Dyer C R.Photorealistic scene reconstruction by voxel coloring[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,1997:1067-1073.
[36] Sinha S N,Pollefeys M.Multi-view reconstruction using photoconsistency and exact silhouette constraints:A maximum-flow formulation[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2005:349-356.
[37] Boykov Y,Lempitsky V.From photohulls to photoflux optimization[C]//British Machine Vision Conference.2006:1149-1158.
[38] Hallert B.Notes on of cameras and photographs in photogrammetry[J]. Photogrammetria,1968,23(5):163-178.
[39] Slabaugh G G,Culbertson W B,Malzbender T.Methods for volumetric reconstruction of visual scenes[J]. International Journal of Computer Vision,2004,57(3):179-199.
[40] Slabaugh G G,Culbertson W B,Malzbender T,et al.A survey of methods for volumetric scene reconstruction from photographs[C]//International Workshop on Volume Graphics.New York,USA:ACM,2001:81-100.
[41] Chhabra V.Reconstructing specular objects with image based rendering using color caching[D]. Worcester,USA:Worcester Polytechnic Institute,2001.
[42] Duan Y,Yang L,Qin H.Shape reconstruction from 3D and 2D data using PDE-based deformable surfaces[C]//European Conference on Computer Vision.Berlin,Germany:SpringerVerlag,2004:238-251.
[43] Labatut P,Pons J P,Keriven R.Efficient multi-view reconstruction of large-scale scenes using interest points,Delaunay triangulation and graph cuts[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2007:420-427.
[44] Szeliski R,Golland P.Stereo matching with transparency and matting[J]. International Journal of Computer Vision,1999,32(1):45-61.
[45] Slabaugh G G,Malzbender T,Culbertson W B,et al.Improved voxel coloring via volumetric optimization[R]. New York,USA:New York University,2000.
[46] Zeng G,Paris S,Quan L,et al.Progressive surface reconstruction from images using a local prior[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2005:1230-1237.
[47] Jin H,Soatto S,Yezzi A J.Multi-view stereo reconstruction of dense shape and complex appearance[J]. International Journal of Computer Vision,2005,63(3):175-189.
[48] Treuille A,Hertzmann A,Seitz S M.Example-based stereo with general BRDFs[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2004:457-469.
[49] Strecha C,Fransens R,Van Gool L.Combined depth and outlier estimation in multi-view stereo[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2006:2394-2401.
[50] Zitnick C L,Kanade T.A cooperative algorithm for stereo matching and occlusion detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(7):675-684.
[51] Furukawa Y,Ponce J.Carved visual hulls for image-based modeling[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2006:564-577.
[52] Starck J,Miller G,Hilton A.Volumetric stereo with silhouette and feature constraints[C]//British Machine Vision Conference.2006:1189-1198.
[53] Lloyd S A,Haddow E R,Boyce J F.A parallel binocular stereo algorithm utilizing dynamic-programming and relaxation labeling[J]. Computer Vision Graphics and Image Processing,1987,39(2):202-225.
[54] Appleton B,Talbot H.Globally minimal surfaces by continuous maximal flows[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(1):106-118.
[55] 张维中,张丽艳,王晓燕,等.基于标记点的图像特征匹配的鲁棒算法[J]. 中国机械工程,2006,17(22):2415-2418.Zhang Weizhong,Zhang Liyan,Wang Xiaoyan,et al.Robust algorithm for image feature matching based on reference points[J]. China Mechanical Engineering,2006,17(22):2415-2418.
[56] 管业鹏,武频,朱晓锦,等.基十视差与基线距相关三目立体匹配法[J]. 仪器仪表学报,2006,27(11):1418-1422.Guan Yepeng,Wu Pin,Zhu Xiaojin,et al.Trinocular stereo matching algorithm based on correlation between disparities and baseline[J]. Chinese Journal of Scientific Instrument,2006,27(11):1418-1422.
[57] Sun C M.Fast stereo matching using rectangular subregioning and 3D maximum-surface techniques[J]. International Journal of Computer Vision,2002,47(1/2/3):99-117.
[58] Cheung G K M,Baker S,Kanade T.Visual hull alignment and refinement across time:A 3D reconstruction algorithm combining shape-from-silhouette with stereo[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2003:375-382.
[59] Goesele M,Curless B,Seitz S M.Multi-view stereo revisited[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2006:2402-2409.
[60] Esteban C H,Schmitt F.Silhouette and stereo fusion for 3D object modeling[J]. Computer Vision and Image Understanding,2004,96(3):367-392.
[61] Taylor C J.Surface Reconstruction from feature based stereo[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2003:184-190.
[62] Furukawa Y,Ponce J.Carved visual hulls for image-based modeling[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2006:564-577.
[63] Fua P,Leclerc Y G.Object-centered surface reconstruction:Combining multi-image stereo and shading[J]. International Journal of Computer Vision,1995,16(1):35-56.
[64] Furukawa Y.High-fidelity image-based modeling[D]. USA:University of Illinois,2007.
[65] Tasdizen T,Whitaker R.Higher-order nonlinear priors for surface reconstruction[J]. IEEE Transactions on Pattern Analysis and Machine Intelligance,2004,26(7):878-891.
[66] Kass M,Witkin A,Terzopoulos D.Snakes:Active contour models[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,1987:259-268.
[67] Domaika F,Ahlberg J.Face model adaptation for tracking and active appearance model[C]//British Machine Vision Conference.2003:326-339.
[68] Van Ginneken B,Frangi A F,Staal J J.Active shape model segmentation with optimal features[J]. IEEE Transactions on Medical Imaging,2002,21(8):924-933.
[69] Lanitis A,Taylor C J,Cootes T F.Automatic interpretation and coding of face images using flexible models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1997,19(7):743-756.
[70] Van Kuilenburg H,Wiering M,Uyl M D.A model based method for automatic facial expression recognition[C]//European Conference on Machine Learning.Berlin,Germany:SpringerVerlag,2005:194-205.
[71] Yuille A L,Hallinan P W,Cohen D S.Feature extraction from faces using deformable templates[J]. International Journal of Computer Vision,1992,8(2):99-111.
[72] Cootes T F,Taylor C J,Cooper D H,et al.Active shape models-Their training and application[J]. Computer Vision and Image Understanding,1995,61(1):38-59.
[73] Cootes T F,Edwards G J,Taylor C J.Active appearance models[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,1998:484-498.
[74] Cootes T F,Taylor C J.Statistical models of appearance for computer vision[DB/OL]. (2004-03-08)[2010-07-26]. http://www.isbe.man.ac.uk/bim/Models/app_models.pdf.
[75] Cootes T F,Edwards G J,Taylor C J.Comparing active shape models with active appearance models[C]//British Machine Vision Conference.1999:173-182.
[76] 山世光.人脸识别中若干关键问题的研究[D]. 北京:中国科学院计算技术研究所,2006.San Shiguang.A number of key issues in face recognition research[D]. Beijing:Institute of Computing Technology,Chinese Adademy of Sciences,2006.
[77] Amstutz E,Teshima T,Kimura M,et al.PCA based 3D shape reconstruction of human foot using multiple viewpoint cameras[C]//International Conference on Computer Vision Systems,Vision for Cognitive Systems.Berlin,Germany:SpringerVerlag,2008:161-170.
[78] Wang J H,Saito H,Kimura M,et al.Human foot reconstruction from multiple camera images with foot shape database[J]. IEICE Transactions on Information and Systems,2006,89(5):1732-1741.
[79] Enright D,Marschner S,Fedkiw R.Animation and rendering of complex water surfaces[C]//Annual Conference on Computer Graphics and Interactive Techniques.New York,USA:ACM,2002:736-744.
[80] Caselles V,Kimmel R,Sapiro G.Geodesic active contours[J]. International Journal of Computer Vision,1997,22(1):61-79.
[81] Baker S,Sim T,Kanade T.When is the shape of a scene unique given its light-field:A fundamental theorem of 3D vision?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2003,25(1):100-109.
[82] Tran S,Davis L.3D surface reconstruction using graph cuts with surface constraints[C]//European Conference on Computer Vision.Berlin,Germany:Springer-Verlag,2006:219-231.
[83] Diebel J R,Thrun S,Brunig M.A Bayesian method for probable surface reconstruction and decimation[J]. ACM Transactions on Graphics,2006,25(1):39-59.
[84] Szeliski R,Zabih R,Scharstein D,et al.A comparative study of energy minimization methods for Markov random fields with smoothness-based priors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2008,30(6):1068-1080.
[85] Brandt S.Maximum likelihood robust regression with known and unknown residual models[C]//Statistical Methods in Video Processing Workshop.Clayton,Australia:Monash University,2002:97-102.
[86] Liu S,Cooper D B.Ray Markov random fields for imagebased 3D modeling:Model and efficient inference[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2010:967-971.
[87] Medina R,Garreau M,Toro J,et al.Markov random field modeling for three-dimensional reconstruction of the left ventricle in cardiac angiography[J]. IEEE Transactions on Medical Imaging,2006,25(8):1087-1100.
[88] Paulsen R R,Hilger K B.Shape modelling using Markov random field restoration of point correspondences[C]//International Conference on Information Processing in Medical Imaging.Berlin,Germany:Springer-Verlag,2003:1-12.
[89] Laurentini A.The visual hull concept for silhouette-based image understanding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1994,16(2):150-162.
[90] Lazebnik S,Boyer E,Ponce J.On computing exact visual hulls of solids bounded by smooth surfaces[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2001:156-161.
[91] Terzopoulost D,Platt J,Barr A.Elastically deformable models[C]//Annual Conference on Computer Graphics and Interactive Techniques.New York,NY,USA:ACM,1987:205-214.
[92] Isidoro J,Sclaroff S.Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints[C]//IEEE International Conference on Computer Vision.Piscataway,NJ,USA:IEEE,2003:1335-1342.
[93] Matusik W,Buehler C,McMillan L.Polyhedral visual hulls for real-time rendering[C]//12th Eurographics Workshop on Rendering Techniques.Berlin,Germany:Springer-Verlag,2001:115-125.
[94] Gargallo P,Sturm P.Bayesian 3D modeling from images using multiple depth maps[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2005:885-891.
[95] Drouin M A,Trudeau M,Roy S.Geo-consistency for wide multi-camera stereo[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2005:351-358.
[96] Pons J P,Keriven R,Fangeras O.Modelling dynamic scenes by registering multi-view image sequence[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2005:822-827.
[97] 孙瑾,顾宏斌.计算机视觉系统框架结构研究[J]. 计算机工程与应用,2004,40(12):44-47.Sun Jin,Gu Hongbin.The research on the system framework of computer vision[J]. Computer Engineering and Applications,2004,40(12):44-47.
[98] Forssen P E,Meger D,Lai K,et al.Informed visual search:Combining attention and object recognition[C]//IEEE International Conference on Robotics and Automation.Piscataway,NJ,USA:IEEE,2008:935-942.
[99] Lowe D G.Towards a computational model for object recognition in IT cortex[C]//IEEE International Workshop on Biologically Motivated Computer Vision.Piscataway,NJ,USA:IEEE,2000:20-31.
[100] Snow D,Viola P,Zabih R.Exact voxel occupancy with graph cuts[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos,CA,USA:IEEE Computer Society,2000:345-352.
[101] Faugeras O,Keriven R.Variational principles,surface evolution,PDE's,level set methods,and the stereo problem[J]. IEEE Transactions on Image Processing,1998,7(3):336-344.
[102] Feng J,Ip H H S.A multi-resolution statistical deformable model(MISTO)for soft-tissue organ reconstruction[J]. Pattern Recognition,2009,42(7):1543-1558.
