Abstract: To protect the privacy and improve the recognition accuracy,a method of color face privacy protection recognition based on double random phase encoding and quaternion Grassmann average networks is proposed.Firstly,three color components of each color face image are encoded into a pure quaternion matrix.Then the quaternion value matrix is performed double random phase encoding in quaternion gyrator domain.In order to conceal the content of color facial image,only a small part of encrypted data is randomly selected by using random binary amplitude mask and preserved for decryption.For the invisible decrypted face images,quaternion Grassmann average networks is employed to extract features and recognition rate is calculated by linear support vector machine.In case that the face template is leaked,alternative one can be reproduced by new random binary amplitude matrix,which makes face template cancelable and keeps original face images in safe.The four datasets were used including Aberdeen,Georgia Tech,Visible Light and YouTube Makeup and the recognition rates are compared with other three face image privacy protection methods,the experimental results have demonstrated that the proposed method can effectively improve the recognition rates and it shows better robustness to the change of dataset.