高级检索
王卓程, 张景峤. 基于三维手部骨架数据的连续手语识别[J]. 计算机辅助设计与图形学学报, 2021, 33(12): 1899-1907. DOI: 10.3724/SP.J.1089.2021.18816
引用本文: 王卓程, 张景峤. 基于三维手部骨架数据的连续手语识别[J]. 计算机辅助设计与图形学学报, 2021, 33(12): 1899-1907. DOI: 10.3724/SP.J.1089.2021.18816
Wang Zhuocheng, Zhang Jingqiao. Continuous Sign Language Recognition Based on 3D Hand Skeleton Data[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(12): 1899-1907. DOI: 10.3724/SP.J.1089.2021.18816
Citation: Wang Zhuocheng, Zhang Jingqiao. Continuous Sign Language Recognition Based on 3D Hand Skeleton Data[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(12): 1899-1907. DOI: 10.3724/SP.J.1089.2021.18816

基于三维手部骨架数据的连续手语识别

Continuous Sign Language Recognition Based on 3D Hand Skeleton Data

  • 摘要: 为有效地消除手语识别过程中背景、光照等干扰因素带来的视觉问题,采用低冗余的骨架数据表达手语信息,设计了一个端到端连续手语识别模型.首先,分别从帧内和帧间提取手型和轨迹特征,可以有效地降低原始样本的离散程度;其次,构建一系列并行的双路残差网络对手型和轨迹特征进行优化与融合,生成时空特征序列;最后,基于注意力机制的编码-解码网络实现时空特征序列到翻译文本的映射.使用Leap Motion收集建立了一个基于三维手部骨架数据的手语数据集LMSLR.实验结果表明,在LMSLR数据集和公共的CSL数据集上,该模型与大多数基于视频处理的模型相比具有较高的准确率和较小的计算量.

     

    Abstract: In sign language recognition,it is necessary to eliminate the visual problems caused by interference factors such as background and light.Therefore,an end-to-end continuous sign language recognition model is designed using low-redundant skeleton data.Firstly,the shape and trajectory features are extracted from intra frame and inter frame respectively,which can reduce the discreteness of the original samples.Secondly,a se-ries of parallel two-stream residual networks are constructed to fuse shape and trajectory features,further gen-erate the spatial-temporal feature sequence.Finally,the attention-based encoder-decoder network is used to re-alize the mapping of the fused feature sequence to the translated text.In addition,a new skeleton-based sign language dataset using Leap Motion is collected named LMSLR.Experimental results on the LMSLR dataset and the public CSL dataset show that the proposed model has higher accuracy and lower computational com-plexity than most models based on video processing.

     

/

返回文章
返回