Missing Joint Point Repair of Sign Language Sample Skeleton Based on Conditional Generation Adversarial Networks

Due to the influence of occlusion, angle of view, illumination and other factors, the detection of skeleton joint points of sign language samples by computer vision technology is usually missing, which leads to the lower accuracy of sign language recognition. To solve this problem, a new method based on conditional generation adversarial networks (CGAN) was proposed to repair the missing joint points of sign language sample skeleton. Firstly, by analyzing the distribution of missing joint points in the incomplete skeleton of sign language samples, the distribution probability model of missing joint points was constructed. Secondly, the missing joint points generated by the distributed probability model are added into the complete skeleton, and uses these incomplete skeletons for the training of the generator and discriminator in the CGAN framework, the generator trained through the CGAN framework can generate skeletons without missing joint points on the condition of the incomplete skeleton. Finally, we use the generated skeleton to fill the incomplete skeleton and complete the repair. Experiments are carried out on the Chinese sign language dataset CSL, after the generator iteratively trained for 80 times, the average root mean square error between the generated skeleton and the complete skeleton is reduced from 0.019 to 0.001. 120 times of iterative training of sign language recognition network was built on the sign language samples with missing joint points repaired, compared with no repair, the recognition accuracy is increased from 90.6% to 99.6%. The experimental results show that the proposed method can effectively repair the missing joints and greatly improve the accuracy of sign language recognition.