Visual Detection for Crawler on Booking Servers

Large losses have been caused by malicious crawlers,demanding an anti-crawler system.This paper presents a general visual analytics system to detect crawlers in airlines’booking servers.First,several data visualization and analysis tools,including route map,histogram and pie chart,are provided to show the result of crawler detection at any time every day.Then,based on SVM classifier and combined with IP address aggregation,an effective algorithm is designed for recognizing various types of crawlers,especially dynamic IP ones.Additionally,by means of feature value filtering and according to IP’s historic behavior,the user can select optimum samples to retrain the SVM classifier.The results of our experiment using the log data from a airline show that our system can identify most crawlers and can adapt to the evolution of crawlers to maintain long-term effectiveness.