Abstract: To employ emotion semantic of associated audio modal data to guide extraction of highlights of video, a method, driven by audio emotion perception, is presented. An audio classifier, based on a binary-tree support vector machine, is employed to obtain the mid-level audio type. With an emotion-mapping model integrated, high-level emotion semantic for associated audio modal data is obtained finally. The complete audio emotion perception model, including an audio classifier and an emotion-mapping model, is a pro-posed to analyze the emotion semantic fluctuation of associated audio. Furthermore, video highlights are extracted with additional aids including a start-stop positioning strategy for highlight and a method for audio video synchronization. Taking emotion semantic fluctuation series of audio as core data, an entire video highlight extraction framework, driven by audio-emotion, is constructed with a two-stage emotion perception model of audio, which completes the most important leading analysis. The experiment demonstrates that the proposed framework can achieve high recall ratio and integrity with good generalized ability in the case of a certain guaranteed accuracy.