NeurIPS 2022

FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction

 

Qiao Feng1, Yebin Liu2, Yu-Kun Lai3, Jingyu Yang1, Kun Li1,*

1 Tianjin University   2 Tsinghua University   3 Cardiff University  

  * Corresponding author

 

[Paper] [Supplemental] [Code]

 

Abstract

The advent of deep learning has led to significant progress in monocular human reconstruction. However, existing representations, such as parametric models, voxel grids, meshes and implicit neural representations, have difficulties achieving high-quality results and real-time speed at the same time. In this paper, we propose Fourier Occupancy Field (FOF), a novel, powerful, efficient and flexible 3D geometry representation, for monocular real-time and accurate human reconstruction. A FOF represents a 3D object with a 2D field orthogonal to the view direction where at each 2D position the occupancy field of the object along the view direction is compactly represented with the first few terms of Fourier series, which retains the topology and neighborhood relation in the 2D domain. A FOF can be stored as a multi-channel image, which is compatible with 2D convolutional neural networks and can bridge the gap between 3D geometries and 2D images. A FOF is very flexible and extensible, e.g., parametric models can be easily integrated into a FOF as a prior to generate more robust results. Meshes and our FOF can be easily inter-converted. Based on FOF, we design the first 30+FPS high-fidelity real-time monocular human reconstruction framework. We demonstrate the potential of FOF on both public datasets and real captured data. The code is available for research purposes.


Method

 

 

Fig 1. The proposed Fourier Occupancy Field (FOF).

 

 

Fig 2. The reconstruction pipeline based on FOF.

 


Results (Twindom)

 

 


Results (THuman2.0)

 

 


Real-Time Reconstruction (Real-World)

 

 


Technical Paper

 


Citation

Qiao Feng, Yebin Liu, Yu-Kun Lai, Jingyu Yang, Kun Li. "FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction". In Proc. NeurIPS, 2022.

 

@inproceedings{li2022neurips,
  author = {Qiao Feng and Yebin Liu and Yu-Kun Lai and Jingyu Yang and Kun Li},
  title = {FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction},
  booktitle = {NeurIPS},
  year={2022},
}