Abstract: Aiming at that the mismatch removal methods cannot effectively handle the ambiguity problem of constraints, this study proposes a novel mismatch removal method based on adjacency matrix updating. Initially, the compatibility between matches is determined using point pair feature (PPF) constraints to construct an undirected graph. Subsequently, the adjacency matrix and voting scores of the matches are computed to rank the matches. The top K matches are selected to update the adjacency matrix, and an annealing technique is employed to gradually reduce the value of K. Ultimately, matches with high voting scores are retained. Experimental results on the UWA3M, augmented ICL-NUIM, and WHU-TLS datasets demonstrate that the proposed method achieves both high recall and precision, with F of 0.882 1, 0.706 9, and 0.872 9, respectively, outperforming comparative methods and validating the effectiveness of the proposed approach.