Abstract: The image semantic segmentation methods always had edge blurring and low accuracy problems. Addressing these problems, this paper proposes a multi-category edge perception method. Firstly, our novel Multi-sigmoid loss function is used for multi-category objects segmentation, which is combined with the pretrained FCN-CRF network on the COCO dataset to build a sematic segmentation model that can optimize the class boundaries. Then, the bottom-up information decoding subnet is added to HED model, the image enhancement algorithm called subpixel is employed to up-sample and fuse features between adjacent scales. Through above methods, we construct the edge detection model called Multi-Scale encode and decode Fusion (MSDF) model. Finally, the segmentation information extracted from FCN-CRF is taken as unary potentials and edge features detected from MSDF are taken as pairwise potential, the global energy function consists of these two components, which is minimized to further optimize the segmentation results. Subjective experiments are performed on the two most common datasets, including Pascal-context and SIFT Flow. The results show that the proposed model performs well, which can be applied to the relevant area such as image sematic segmentation and salient object detection.