Abstract: In order to deal with the difficulty of traffic sign image detection and recognition under different illumination condition, an illumination image enhancement algorithm based on Retinex-Gamma is proposed. The algorithm is combined with Mask R-CNN, which is called Retinex-Gamma Mask R-CNN algorithm. Firstly, based on the illumination reflection imaging model, the image RGB space is transformed into HSV space, the V channel is processed by multi-scale Gaussian filtering to obtain the illumination component, the illumination component is used to extract the reflection component, and the reflection component is linearly optimized. Secondly, the two-dimensional Gamma function is adjusted by using the distribution characteristics of illumination components, and the optimized brightness components are obtained. Finally, the enhanced V channel is obtained by using the mixed space enhancement method to reconstruct the image. The ZCTSDB dataset used for the experiment has a total of 15 724 images and contains driving environments with different lighting. The experimental results show that compared with the standard Mask R-CNN, the average accuracy of Retinex-Gamma-Mask R-CNN algorithm for target detection of traffic signs is improved by 0.161%, and the average accuracy of instance segmentation of traffic signs is improved by 0.363%.