Abstract: In the process of automatic report generation for medical images, due to the small sizes, irregular shapes of lesions, and small amount of training data, it was easy to lead to misdiagnosis and missed diagnosis in the reports. This paper proposes a medical report generation method based on multi-scale feature fusion and cross-training. Firstly, the method combines the coarse-grained features after conditional global pooling with the fine-grained features after random discarding to enhance the model’s perceptive ability to different lesion scales. Secondly, a two-way cross-training strategy through overall data and local details is used to indirectly enrich the dataset and improve the robustness of the model, while adapting channel separation principle to better mine channel information separately in the two ways. Finally, an accurate image report is obtained through the multi-head attention encoding and decoding network. Compared with other methods, the scores of METEOR and BLEU-2 are improved by 5.70% and 3.13% on the IU-X-Ray and MIMIC-CXR datasets, respectively. The results show that the proposed method can effectively improve the readability and accuracy of generated reports.