Abstract: This paper proposes a user-assisted intrinsic image decomposition method based on a single RGB-D image. To encode spatially varying illumination, an intrinsic image model which separates the shading image into a distant illumination component and a local illumination component is firstly introduced; then, a new illumination estimation method based on the spherical harmonics is developed to obtain an initial shading image. To improve the accuracy of the decomposition result, the recovered illumination is adopted as a prior of the decomposition and an energy function is built to calculate the distant illumination component; finally, we design an algorithm to efficiently calculate the local irradiance and reflectance components by letting users indicate pixels with similar albedo and different shading on the image. Experimental results demonstrate the validity and flexibility of our approach.