Abstract: A coarse-to-fine estimation method for spatial layout is presented to effectively label the layout relationship of indoor scenes. Firstly, the adaptive threshold detection method with local discontinues is exploited to acquire the long straight lines of the given scene, which are splitted into the vertical lines and horizontal ones in terms of the corresponding directions. The vertical and horizontal vanishing points are estimated based on the vote mechanism and orthogonality principle, and the pairs of the rays led from two vanishing points at equal angular interval are used to generate the candidates of the given scene layout. Next, the informative edge and geometric context of the given scene are estimated with VGG-16 full convolution neural network, and the softmax classifier is applied to deciding the given fc7 features to obtain the layout category, while the global features merged with the informative edge and layout category are generated to roughly select the layout candidates. Then, the normal vector and depth map of the given scenes are estimated with the VGG-based spatial multi-scale convolution neural network to extract the related normal vector and geometric depth feature. And next, the 3D box spatial layout model can be parameterized by the angles between the rays from vanishing points, while the line membership, geometric context, normal vector and depth feature are accumulated via geometric integral image to extract the regional features of layout candidates, and the structural model parameter can be learned with cutting-plane method. Finally, the layout candidate with the highest structural prediction score is selected as the final spatial layout. Experimental results on the Hedau and LSUN datasets demonstrate that the presented method can obtain more accurate number of divided polygons and more precise boundary positions of spatial layout.