Abstract: Large language models have gained prominence due to their outstanding task-solving capabilities. The evolution from basic language modeling and text generation tasks to complex reasoning tasks has facilitated the transition of large language models from general to specialized capabilities. This gradual implementation across various application scenarios underscores their utility in interaction with users. Despite the unprecedented and profound impact of these models, they are often criticized for their lack of transparency in internal mechanisms and ethical considerations. There remains a necessity for more explainability researches to unveil the mystery of large language models, thereby enhancing their abilities to adapt to downstream tasks and improving user experience. The outputs of large language models are influenced by both the interactive process and the strict norms of domain applications. This paper innovatively proposes a comprehensive review of explainability studies of large language models from two dimensions: model and user. Specifically, we combine model process-transparency and interaction-controllability with the credibility of model outputs. First, we explore the model itself, categorizing existing interpretative methods based on internal and external techniques for training fine-tuning and enhancement. Next, from the perspective of human-computer interaction, we discuss research on guiding model decisions through user-input prompts to enhance model explainability. Finally, we outline the limitations faced by explainability studies of large language models and offer prospects for future development.