Abstract: Recently reported lightweight networks have promoted the application of real-time semantic segmentation on mobile platforms. However, the linear combination operation performed in lightweight networks do not consider the relationship between fused features, resulting in limited segmentation accuracy. To solve this dilemma, a lightweight network with convolutional attention feature fusion based on encoder-decoder architecture is proposed in this paper. In the encoder, a dilated MobileNet block is given based on MobileNetv2 to create sufficient receptive fields and enhance representation ability of the lightweight backbone. In the decoder, convolutional attention feature fusion module is given. Relative attention weights that contain interactions between channel, height and width are used to aggregate feature maps. Specifically, without a pretrained model, postprocessing or extra data, the lightweight network with convolutional attention feature fusion has only 0.68 million parameters and achieves a 72.7% mean intersection over union on the Cityscapes dataset with a speed of 86 frames per second and a 67.9% mean intersection over union on the Camvid dataset with a speed of 105 frames per second on a single 2080Ti GPU. The comprehensive experiments demonstrate that our model achieves favorable trade-off between accuracy, model size and speed.