Research on Monocular Depth Estimation Based on Audio-visual Fusion Coupling Improved Coordinate Self-Attention in Indoor Environment

Aiming at the characteristic that both of monocular image and sound echo signal contain spatial information, this paper proposes a monocular depth estimation method based on audio-visual fusion. The proposed approach comprises an interval classification network and an interval probability distribution estimation network. The interval classification network takes the acoustic echo signal and material features as input, while the interval probability distribution estimation network takes the monocular image as input. Classification and probability distribution of depth intervals are combined linearly to construct the final depth map. Pyramid pooling module couples of acoustic echo and material information. Depth interval probability distribution prediction adopts the encoder-decoder structure. In order to effectively extract local and global information, this paper uses the method of combining convolutional neural network and Transformer in the encoder stage. In the decoder stage, this paper improves the coordinate attention module and proposes a coordinate self-attention module. The experimental results show that the audio-visual multimodal fusion analysis has achieved competitive results on both the Replica and Matterport3D datasets. Ablation experiments indicate that the proposed coordinate self-attention module can improve the quality of depth estimation.