Abstract: According to the high dimensional structure of the video modality, this paper formulates the motion saliency extraction as the tensor based low-rank recovery and block-sparse representation problem.First, the proposed approach utilizes the accelerated proximal gradient based tensor recovery to reconstruct the low-rank and sparse sensors of RGB color channels, through which the rough motion saliency can be initially separated from the background.Then, the sparse tensors of the three color channels are grouped together and unfolded into the matrix in terms of the frame number.Subsequently, the matrix recovery method is further employed to process this unfolded matrix such that the small but irrelevant sparse components can be removed.Finally, the adaptive threshold method is utilized to select the block-sparse mask with respected to motion saliency and the holes within the mask is simultaneously filled, whereby the motion saliency can be well extracted.Comparing with traditional methods, the proposed approach utilizing the tensor model to preserve the spatial structure of the video modality, not only can reduce the missing detection problem, but also isable to remove the interferences of dynamic background.The experimental results have shown the promising performances.