Abstract: Texture filtering with structure-preservation is a primary pre-processing method in many applications of computational photography and image analysis. Conventional methods may fail and generate blurred edges when the input image contains textures with strong contrast, since they just consider local contrast information of an image. Therefore, we propose a non-local joint texture removing method based on patch shift. First, a guidance map which preserves edges while suppressing texture is acquired via patch shift. Then a non-local joint texture filtering is applied with the assistance of the guidance map. By utilizing the similarity between non-local patches, we further suppress the blur caused by the guidance map and extract distinct image structures. We show that our method outperforms previous methods in removing texture while preserving primary image structures through a large number of experiments. Moreover, we perform the edge detection and detail enhancement on the images after texture removal and illustrate that our method benefits such application scenarios.