Abstract: The modality difference between visible light and infrared images is the fundamental reason which increases the difficulty of shared feature extraction for the visible-infrared re-identification. To reduce the difference between the two modalities and improve recognition performance, this paper proposes a progressive difference reduction network (PDRNet). Firstly, in the retrieval stage from visible light targets to infrared image sets, a fact-specific intervention module is designed based on causal reasoning theory. The intervention image generated through channel transformation is used to intervene in visible light images. It suppresses the color information interference in visible light images. Secondly, in the retrieval stage from infrared targets to visible light image sets, a channel coordination module is designed to convert multi-channel feature extraction into a single channel approach, focusing the model on learning the channel correlation between visible and infrared images. Finally, to achieve mutual retrieval between visible light and infrared target images, a modal balance loss is designed. By intervention images, visible light images, and infrared images, multiple modality balance learning is completed, further suppressing the color feature and compensating for identifiable lost features of visible light images during fact-specific intervention processes. The simulation experimental results show that PDRNet has achieved good performance on both SYSU-MM01 and RegDB benchmark datasets. Compared to existing the state-of-the-art methods, rank-1 and mAP has been improved by more than 2 percentage points, respectively. Model source code:https://pan.baidu.com/s/15LyVjCT-KEqaGoOc4nRlBw?pwd=0226.