Abstract: In the object tracking based on deep learning, the directional perturbation algorithm is proposed to solve the problem that the tracking accuracy, stability and success rate will be seriously affected when the object moves rapidly, the camera is offset and the object is lost. This paper makes full use of the characteristic that the convolution neural network can locate to change the perturbation center of the particle filter and to make directional disturbance sampling. It makes the candidate sample closer to the real location, accelerates the object recovery and prevents the object from losing, thus improving the precision and success rate of the tracker. At the decision stage, the location hot spot map is obtained first. Then, the HOG features of the last frame result position and the next frame candidate position are extracted respectively. Finally, the optimal solution is found by calculating the similarity. After adding the HOG feature, the tracker can adapt to more complex scenes and improve the robustness of the tracker. Experimental results with FCNT, MEEM and other methods on obt-13 benchmark database show that the proposed algorithm can effectively improve the tracking accuracy, success rate and robustness under the condition of small resource consumption. The proposed algorithm can be better applied to the actual scene and can be extended to other trackers.