Abstract: With filters based on low-level vision, it is challenging to remove multi-scale texture accurately and avoid overblurred weak edges in the texture-structure decomposition. Aiming at the problem, this paper proposes a method which combines semantic edge detection with directional total variation for the decomposition task. Firstly, a network with richer convolutional features is used to extract multi-scale edges of different objects and estimates edge confidence for each pixel. Then, texture confidence of each pixel is estimated from the directional total variation, and the patch shifting algorithm is used to smooth out the strong texture edges which are often misclassified as structure. Finally, the edge confidences are incorporated into texture confidence to weight the decomposition model. Experiments are conducted on the three datasets, which are BSDS500, NYUD and RTVD, and the results demonstrate our method produces more accurate structure layers and provides better visual quality compared to state-of-the-art of decomposition methods. The GPU-based implementation of our method is fast even for high-resolution images.