Geometry-guided Dense Perspective Network for Speech-Driven Facial Animation


Jingying Liu†1, Binyuan Hui†1, Kun Li*1, Yunke Liu1, Yu-Kun Lai 2, Yuxiang Zhang3, Yebin Liu3, Jingyu Yang 1

1 Tianjin University   2 Cardiff University   3 Tsinghua University

Equal contribution  * Corresponding author



Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model.


[Code] [Paper]



Fig 1. Method overview.







Fig 2. Qualitative evaluation results for clean audio inputs.




Fig 3. Qualitative evaluation results for noisy audio inputs.



Fig 4. Animations across languages.



Fig 5. Animations across sentences.


Technical Paper



Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu-Kun Lai, Yuxiang Zhang, Yebin Liu, Jingyu Yang. "Geometry-guided Dense Perspective Network for Speech-Driven Facial Animation". IEEE Transactions on Visualization and Computer Graphics 2021.


  author = {Jingying Liu and Binyuan Hui and Kun Li and Yunke Liu and Yu-Kun Lai and Yuxiang Zhang and Yebin Liu and Jingyu Yang},
  title = {Geometry-guided Dense Perspective Network for Speech-Driven Facial Animation},
  booktitle = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},