Abstract: With the development of wearable devices,it is great research value to conduct human behavior detection based on wearable sensor data.At present,most human behavior recognition work is based on im-ages.However,there are two challenges in using computer vision technology for human behavior recogni-tion.Firstly,it is difficult to make users involved in the data collection in the nature under the true state of motion data,before starting data collection,it is often necessary to train the personnel involved in data col-lection and strictly regulate their collection actions.Data that collected in that way will be a departure from the real life.Its research value will be discounted.Secondly,the privacy protection of data collectors is in-volved in the process of data collection.To this end,a data feature extraction algorithm is proposed based on deep learning.Firstly,introduces the branch structure of neural network on the basis of flexible convolution kernel setting to extract the depth characteristics of the original data at multiple scales.After that,the data features obtained by each branch are fused and used as the input experiment of the next convolution layer.The experimental results show that compared with the current mainstream algorithm, proposed algorithm has achieved better results in accuracy and recall rates on the three standard data sets of MHEALTH, WHARF and USCHAD. In addition, the method has been verified on two newer data sets, Stanford-ECM Dataset and DATAEGO Dataset, and the results show that the method has good generalization ability.