Abstract: Currently, retrieval-augmented generation frameworks have increasingly become an important technology for enhancing the performance of large language models. By integrating various external knowledge sources, they effectively mitigate the issues of outdated knowledge and lack of domain-specific information in large language models. However, challenges persist in aligning retrieved knowledge with user needs and evaluating how external knowledge contributes to generated content within these frameworks. To address this, we introduce KLMVis—a visual analysis system designed to optimize the process of knowledge graph-based retrieval augmentation and to improve the interpretability of language model outputs. This system offers an interface allowing users not only to explore knowledge graphs but also select relevant evidence and engage actively in the retrieval process. Additionally, with its multiple visualization components, it aids users in making more prudent evidence selections while deepening their understanding of the model's decision-making logic. Finally, upon conducting two usage scenarios and a user study on two distinct knowledge graphs, we demonstrate that KLMVis excels in improving retrieval efficiency, assisting users in evaluating the augmented outputs.