Landmark-Based Facial Expression Recognition by Joint Training of Multiple Networks

Information contained in expression image sequence is more abundant than single expression image, thus expression recognition based on the former is easier to achieve better results. An expression recognition method based on facial landmark information and joint training of two deep neural networks is presented in this paper. Firstly, fixed number of frames which maximize the distance among them were extracted from variable length image sequence. Then, coordinates of landmarks were extracted for preprocessing. Next, microcosmic deep network(MIC-NN) and macroscopic deep network(MAC-NN) were trained independently using landmark information. Finally, a loss function which punish the differences between MIC-NN and MAC-NN was applied for joint training of them, and their fusion network(FUS-NN) was tested as final prediction model. Experiments on CK+, Oulu-CASIA and MMI database indicate the recognition rate of FUS-NN surpass most of known methods by 1%-15%, only lags behind the optimal model by 2%in MMI database. However, the time complexity of FUS-NN is sharply reduced compared to those models with similar performance, achieving better balance between recognition rate and computing resources.