Abstract: The Chinese painting poem is a very special art form in the history of world art.It combines ancient Chinese literature and fine arts,complements each other and blends together.In order to obtain computer-based painting poetry,an automatic poetry generation is proposed based on ancient Chinese paintings.It extracts multiple sentences from ancient paintings,which improves the literary expression ability of ancient poems in paintings.Firstly,a multi-sentence annotation data set for ancient paintings is established,and then semantic features of ancient paintings are extracted through an improved image captioning method.Finally,these modern text descriptions are converted into a four-character poem through a two-way LSTM encoding and decoding framework.The experiment on the paintings of the Song Dynasty demonstrates that the coherent and prosodic poems generated by our method are consistent with the original content and context of the ancient paintings.User study shows that the content consistency and user satisfaction of our method are better than keyword-based methods,which proves the validity of the proposed method。