Abstract: Chinese Calligraphy plays an important role in Chinese culture.The simulation of real brush painting is an open issue in the field of computer graphics.Traditional methods represent strokes as simple geometric structures.It lacks change of shape because of the difficult modeling processes,which makes it difficult to simulate the real writing strokes of a brush.This paper proposed a computer vision-based method that took image data as input through four cameras.First,we built a Deeplabv3+-based algorithm by optimizing the poor segmentation performance in small size categories in order to extract the key information such as the brush.Then,the method obtained the relative pose of the penholder by the Hough transform and the PnP pose estimation.Next,the method corrected and fused the stroke images based on the pose information.We proposed an algorithm to estimate unknown regions that were not obtained by cameras.We built a dataset containing more than 400 images under various Calligraphy scenarios.The experiments of segmentation algorithm showed that our method has better mean intersection-over-union(mIOU)of 0.849,improved by 0.117 compared with the baselines.Especially in the small categories,the intersection-over-union(IOU)reached 0.59 and improved by 0.473.The experiments of final stroke generation showed that our method can produce much more realistic strokes in real-time without complex brush models.