Abstract: Pan-sharpening, as one of the most commonly used techniques in current remote sensing applications, aims to inject spatial details from Panchromatic (PAN) images into Multispectral (MS) images to obtain high-resolution multispectral images (HRMS). However, current Pan-sharpening methods usually require the paired PAN and MS images as input, which limits their usage in some scenarios. Besides, excessive introduction of spatial details from PAN images can also cause HRMS artifacts and spectral distortion. To address these issues, this paper first develops a novel representation paradigm for the spatial detail injection model and then propose a memory-based spatial detail (MSDN) network based on this paradigm. The network can preserve spatial details from PAN images during the training and reconstruct the corresponding spatial details according to the MS features during inference. Finally, the proposed MSDN is integrated into existing CNN-based Pan-sharpening framework for experimental validation. The experimental results on the public satellite datasets show that the proposed method improves the ERGAS index by 1.2 compared with the baseline method. Furthermore, compared with other state-of-the-art methods, our framework also has significant advantages in image quality assessment.