Abstract: Content-based classification of sports video is one of the critical steps in the efficient management of a large number of sports video data.To improve the accuracy and generalization ability of sports video classification, a new sports video classification method based on the combination of marked genre shots and bag of visual words model is proposed.Firstly, the definition of the marked genre shots is given, and the video frame training database of marked genre shots is constructed with the marked genre shots.Secondly, the pyramid visual word bag model is constructed based on the video frame training database, each video frame is represented with a visual words frequency vector, and then the SVM is used to classify the video frame.Subsequently, by analyzing the misclassification causes, the isolated frame removal algorithm is proposed to eliminate the representative frame misclassification.Finally, as the sports video, according to its combination type, can be divided into single sports video and mixed sports video, two different classification algorithms for single sports video and mixed sports video are proposed.Experimental results show that our method has the advantages of simple implementation, fast processing speed and high accuracy.