Abstract: This paper addresses the tracking drift problem in deformable target tracking, which is caused by the target segmentation and modeling with color features. To this end, we propose a novel tracking method based on saliency segmentation and target detection. We utilize graph-based manifold ranking to obtain salient and high-quality target pixels, and quantify these pixels by color and gradient. Then, the quantized values of target pixels and the relative position to the center of the target are used to represent the target model, which is used for tracking target by detecting the target center. A drifted constraint based on the global color features is used to correct the central position drift caused by the continuously deforming the target. Moreover, the tracking results are adopted as the constraints of saliency segmentation and global color features, and the target model is updated according to the segmentation results. The experimental results demonstrate that the proposed method is able to improve the precision and stability for deformable target tracking.