Abstract: In this paper,an algorithm of tracking and detecting planar surface boundaries from videos is proposed for architecture scenes.The algorithm was divided into three steps.The planar surface boundaries were given for the first frame.In the first step,the homography matrix between the current frame and its previous frame was used to track planar boundaries.Considering that the tracking process inevitably generates cumulative errors,the RANSAC algorithm was used firstly to fit the lines on planar boundaries,and then the absolute quadratic curve was incorporated as a constraint and finally the LM algorithm was applied to optimize line segments.The second step was to detect the newly appearing planar surfaces.Assuming that the current planar boundaries are composed of two different groups of parallel line segments,the third group of parallel line segments outside the current planar boundaries was extracted,and then the newly appearing planar boundaries were detected.Finally,after detecting the newly appearing planar boundaries,all the planar boundaries of the current frame were tracked to the next frame.When the newly appearing planar areas increased or decreased,the line segments on the planar boundaries were updated by combining the Flood-Filling algorithm and image segmentation based on the assumption that reflectance are similar on the same plane.Experimental results of five realistic scenes show that the proposed algorithm can detect and track the planar surfaces accurately and robustly.