Unified Multi-Category 3D Object Generation from Sketches

Existing sketch-based 3D modeling deep neural networks are typically trained separately for each object category, resulting in poor generality. However, jointly training the modeling networks for multiple categories leads to category confusion and deficiency of shape details. Based on an unsupervised sketch-based 3D model generation framework, a 3D model generation network, MC-SketchModNet, is proposed for joint training of multi-category objects without losing shape details. First, by introducing a sketch category embedding branch, the sketch features and the object category prior are effectively fused to synthesize high-quality 3D models in the generation procedure. Secondly, more views are added to supervise the generation network training as they provide more constraints for 3D shapes from coarse sketches in the training process, thus eliminating the category ambiguity. The experimental results on both synthetic sketches from ShapeNet and freehand sketches demonstrate that compared with the baseline method multi-class trained SoftRas, the proposed MC-SketchModNet can effectively eliminate the category ambiguity in the generated models and obtain 3D models with higher quality and richer details, having an increase of 4.91% in voxel IoU. Moreover, by introducing the object category embedding, interactive category control is supported to specify particular object categories when the input sketch is ambiguous in semantics.