Abstract: In order to improve the precision and efficiency of the motion sensing interaction algorithm based on projection speckles, a fast method for recovering the depth information is proposed by using the zero-mean normalized cross correlation（ZNCC） operator. The gesture classifier is constructed by introducing a hierarchical decision tree of feature selection. Firstly, the disparity of speckle images is obtained during the 3D scene reconstruction by using ZNCC. The depth images of the target scenes are calculated through triangulation theory, and the redundant computation problem of ZNCC is solved by employing the GPU parallel architecture and improved calculation formula. Then, an improved random forest algorithm is applied to complete the classification of body parts during the pose estimation. The mean shift algorithm is used to extract the joint and skeleton points. The feature dimension is reduced by removing the invalid features through a feature selection method. Furthermore, the prediction accuracy is improved via hierarchical decision in the combination of decision trees according to the features of body parts and postures. The overall integration of the system is realized, and the effectiveness of the proposed methods is verified by experiments.