Abstract: The end-to-end audio classification method based on transformer is proved to be better than two-dimensional convolution in multiple scenes. In view of the current popular audio recognition method based on serialization learning transformer, which focuses on the importance of current features in time sequence, and the lack of feature description of simultaneous sequence, a method of parameter-free attention combined with self supervised feature construction is proposed to further improve audio classification. In this method, the parameter-free attention mechanism is constructed in the simultaneous order feature to fit the multi-local extreme value distribution. At the same time, in the process of model learning, the input spectrum is randomly masked in time domain and frequency domain, and self-supervision information is added to effectively learn the audio spectrum details and classification information. The experimental results using audio set, esc50 and Speech Command show that the accuracy of algorithm in this paper improves by 0.46%~1.20%, compared with the current state of the art method.