Abstract: Gray world algorithm typically shows poor illuminant estimation performance due to scene variations of natural images. To address the above problem, an improved gray world algorithm guided by scene semantics is proposed. Firstly, dense-SIFT descriptors are calculated using grayscale images to avoid the interference of color-biased images, followed by the bag of words (BoW) model to generate unordered visual vocabulary. Secondly, based on the average brightness of these visual words, word frequency histograms are constructed weighted by brightness using the spatial pyramid matching (SPM) algorithm. Thirdly, the scene semantics similarity between images employing the histogram cross kernel function, and retrieve a set of candidate images that similar to the test image is calculated. Finally, after removing the outlier images using the isolated forest algorithm, the pixel statistics distributions of all remaining images are used to adaptively infer and update the fixed assumption of the gray world algorithm. The improved gray world algorithm implements color constancy through illuminant estimation. Experimental results in three publicly available color constancy datasets (ColorChecker, Cube+ and NUS) show that the proposed algorithm outperforms comparable improved gray world algorithms in single camera tests (nearly 20% angular error improvement), at the same time, achieves optimal performance in cross-camera tests.