Abstract: We proposed a new data-driven method to infer depth information from a single facade image. A facade is firstly segmented into several regions. By exploiting the symmetry characteristics of facade elements（i.e., windows）, we segment the facade image using a Markov random field（MRF） formulation. We represent each facade by a graph, in which each graph node represents a segmented image region with consistent appearance, and each graph edge encodes the spatial relationship between two distinct image regions. Then we generate a semantic label for each region by automatically matching the graph with our predefined templates in the database. Finally, we perform a global optimization process to produce the final facade model. Experiments demonstrate that our approach can generate favorable results.