Abstract: The selective attention mechanism of human visual system can quickly locate salient targets in the image without training in complex scenes. Combined with the prior information of flame, the saliency based algorithm of quaternion discrete cosine transform is proposed to detect the flame in video. Firstly, two color feature maps were obtained by improving the process of flame color feature extraction according to the special relationship between R,G and B of the flame color in the RGB space. Secondly, the texture feature map was generated by computing the distance of LBP feature vectors. Thirdly, the motion feature map was provided by calculating the zero-crossing frequency according to the dynamic texture characteristics and flicker frequency of the flame. Finally, the four feature maps are regarded as all components of a quaternion and the flame saliency map is produced by quaternion discrete cosine transform with the four feature maps. The experimental results on the flame video library of Bilkent university show that the proposed method is obviously superior to the corresponding algorithms on the accuracy and robustness.