Abstract: Due to the problem of low accuracy and robustness of traditional hand gesture recognition,we propose a hand gesture recognition algorithm based on Faster R-CNN,which is evolved from convolutional neural network(CNN).First,we adjust the key parameters of the Faster R-CNN framework to realize the purpose of detecting and recognizing the gestures at the same time.Second,to improve the recognition accuracy of gesture recognition,we propose DisturbIoU algorithm to prevent the network training from over-fitting.We evaluate the improved algorithm on the public NTU-Microsoft-Kinect-hand posture(NTU)dataset and vision for intelligent vehicles and applications(VIVA)dataset.The experimental results show that the proposed algorithm based on Faster R-CNN can effectively avoid the over-fitting problem of the training model,and obtain a better performance in both accuracy and robustness compared with other existing algorithms.