Abstract: The visual dictionary size is an important factor that affects the performance of scene classification. The large capacity of visual dictionary can influence the classification efficiency due to the lager calculation, while the small capacity of visual dictionary can reduce the classification accuracy because of the influences of polysemy. To solve the problem, a generating method of visual dictionary based on the concept lattice hierarchy is proposed in this paper. First, the initial visual dictionary of training images on bag-of-visterms model is generated. Then, with the use of concept lattice’s hierarchy analysis, the different granularities of reduced visual dictionaries are extracted from the concept lattice by setting different extension thresholds. Finally, the polysemy is deleted by making XOR operations on all types of the reduced visual dictionaries, and a visual dictionary for better describing the image content is generated. Experimental results show that this method is effective.