Abstract: This paper proposes a novel method about semantic expression definition and optimization. The proposed method solves the real-time video-driven facial retargeting problem. First, it defines a set of semantic values, which represents the expression, from sparse 2D feature points with noise. Then, an extended optimization solved in semantic space is employed to improve the quality of expression retargeting. The method neither needs to calibrate camera, nor requires user-specific training to construct the special 3D model and blendshape. So, the proposed algorithm can be applied widely for retargeting expression of network video, real-time social networks, and so on. The experiments demonstrates that the proposed method can achieve accurate and stable facial animation results with the pitch and yaw ranged from -15° to 15°.