Facial pore feature is one of important indicators for face recognition and skin defects detection. In order to eliminate the effect of other skin features from facial pore detection processing effectively, we proposed a new facial pore detection algorithm that coalesces the characteristics of skin pigment distribution and optimal scale. Considering the dissimilarity of skin pigment distribution on different pigment layers, reasonable thresholds were set for SURF and SIFT algorithms to detect the skin features on different pigment layers, which was based on the significant difference of skin features and K-means clustering method. Then, the Euclidean distance was calculated to describe the positions similarity of the same detected points on different layers. Last, the optimal scales were set as thresholds to screen off the interferences of skin features except pore. On this basis, an evaluation criterion of skin pores roughness was also established by using the optimal scales obtained during SIFT process. We selected positive face images from Bosphorus dataset for comparative experiments about pore detection and evaluation. The experiment results show that the proposed algorithm improves accuracy of pore detection, and the evaluation criterion is stable and effective.