To further improve the accuracy of semantic segmentation of low-quality X-ray images of weld defects and reduce the subjective impact and time-consuming of artificial design networks, a semantic segmentation method of weld defects using multilevel and multi-scale neural network self-search is proposed. Through the design of multi-scale lightweight candidate operations, channel attention mechanism and multi-level dynamic network, the expression ability of the network to extract defect features of low-quality images is improved from different dimensions; At the same time, through the exploration of the correlation between the recognition performance of the early and final stages of network training, a gradually fast neural architecture search method using fixed sampling to gradually determine the optimal candidate operation is proposed. In the architecture search phase, 483 X-ray weld defect images collected and annotated by oneself were used for architecture optimization through random cropping, rotation, translation, and other data augmentation operations. Finally, a semantic segmentation network for weld defects was automatically constructed at a lower search cost. Experiments show that using the above ideas for semantic segmentation of X-ray weld defects, the final mIoU index reaches 49.23%, which is higher than 45.41% of the artificial design network and 28.86% of the direct use of model transfer. The self-search speed and segmentation effect of the network are significantly improved.