Abstract: For recognition of massive scenes,it can improve rate of recognition to rectify the building images with perspective distortion.The robustness of existing perspective distortion rectification methods is not enough to be used for complex outdoor building images with strong interference.In this paper,an improved perspective distortion rectification method was given.For the shape features of buildings,the gravity sensor on mobile phones was used in this method and make improvements on Hough transformation.Then RANSAC is used to get rid of some wrong lines.So the robustness of the method was improved.The horizontal and vertical vanishing points of buildings were used to restore affine structure of buildings and make them closer to the training images.In experiment,the rectified images were used to recognize images.Furthermore,a fusion strategy with mutual information was proposed to combine the result of a side view and a rectified image.The experiments showed that the proposed method can rectify outdoor building images robustly,and greatly improve correct rate of the recognition of outdoor buildings.