3D Human Body Reconstruction with Shadow

Body shadow cast by lights on the ground can provide constraint information for human body reconstruction, which helps to alleviate the depth ambiguity and self-occlusion issues in a single image. In this paper, a deep neural network model based on shadow input is proposed for reconstructing 3D body mesh. First, two independent convolutional neural networks are constructed to respectively extract image feature and shadow feature, and then estimate corresponding body model parameters. Then, a differentiable projection operator (DPO) is formulated to align the outputs of two networks. The loss function built on DPO is applied in two scenarios, namely fine-tuning the neural network and directly refining reconstruction results. Experiments performed on Human3.6M with synthetic shadow showed that mean per joint position error of the fine-tuned network decreased from 1.4 mm to 4.9 mm compared with baseline methods. A small-scale dataset of 3 actors and 2 views with the real shadow is captured. Qualitative results on this dataset demonstrated that directly refined body pose better matches the original image.