Abstract: Since current gesture recognition algorithms are influenced by rotation, translation and scaling, and which can lead to lower recognition rate, this paper proposes a gesture recognition algorithm which is based on the main direction of gesture and Hausdorff-like distance template matching. Firstly, we converted the segmented gesture image to standardized image and calculated the main direction of gesture in the standardized image. Then, we built a 2D rectangular coordinate system to extract the gesture features. Next, we used the method of hand coordinates distribution features to preliminarily recognize the gesture. Finally, the thought of Hausdorff-like distance was used to recognize the final gesture. Experimental results show that this algorithm can achieve real-time correct gestures recognition in relatively stable light conditions. The overall recognition rate can reach 95% and the recognition rate of rotation gestures is more than 90%.