High-Fidelity and Freely Controllable Talking Head Video Generation

Microsoft Research

PEChead generates high-fidelity head pose and expression editing results.


Talking head generation is to generate video based on a given source identity and target motion. However, current methods face several challenges that limit the quality and controllability of the generated videos. First, the generated face often has unexpected deformation and severe distortions. Second, the driving image does not explicitly disentangle movement-relevant information, such as poses and expressions, which restricts the manipulation of different attributes during generation. Third, the generated videos tend to have flickering artifacts due to the inconsistency of the extracted landmarks between adjacent frames.

In this paper, we propose a novel model that produces high-fidelity talking head videos with free control over head pose and expression. Our method leverages both self-supervised learned landmarks and 3D face model-based landmarks to model the motion. We also introduce a novel motion-aware multi-scale feature alignment module to effectively transfer the motion without face distortion. Furthermore, we enhance the smoothness of the synthesized talking head videos with a feature context adaptation and propagation module. We evaluate our model on challenging datasets and demonstrate its state-of-the-art performance.

Video Results

Using PECHead you can create great visual results for head pose and expression editing, including portrait reconstruction and reenactment, and more.

Head Pose Free Editing

Portrait Frontalization

Same-Identity Video Reconstruction

Cross-Identity Video Face Reenactment

Facial Expression Transferring

Video Face Reenactment on Wild Identities


  author    = {Gao, Yue and Zhou, Yuan and Wang, Jinglu and Li, Xiao and Ming, Xiang and Lu, Yan},
  title     = {High-Fidelity and Freely Controllable Talking Head Video Generation},
  journal   = {CVPR},
  year      = {2023},