Abstract: The L₁ trackers are robust to moderate occlusion but computationally expensive, and the oversimplified descriptors are prone to be drift by the noise. To solve these problems, an optimized sparse representation tracking algorithm based on memory gradient pursuit is proposed. Firstly, effective block covariance descriptors are manipulated to represent appearance model. On the basis of the descriptors with templates updated strategy, the robustness to complex scenes and model drift is improved. Moreover, an adaptive scaled unscented transform method with lower computation cost is adopted to approximate the covariance matrix. Then, similarity metric of covariance descriptor is transferred from manifold to Euclidean space. The algorithm takes advantages of fast and stable convergence of memory gradient algorithm to reduce the reconstruction time of sparse coefficient. After that, sparse coefficients are achieved. The experimental results show that the proposed algorithm outperforms other five algorithms.