Abstract: Face image inpainting based on the convolutional neural network has enabled a variety of applications, ranging from criminal investigation to cultural relics protection. However, the results of existing methods are often limited to insufficient diversity and still far from realistic. In this work, we generate more reasonable missing facial content with a variant of variational auto-encoder, as well as generative adversarial network. Furthermore, we impose constraints on latent variables to encourage the distribution of representations to be factorial that making them independent across dimensions. Latter, the optimal boundary was obtained through dynamic programming, and finally we get the seamless results by Poisson image editing. Experiments on CelebA dataset demonstrated that the proposed method achieved better inpainting results and disentanglement.