## Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation > [!Abstract]- > Large Language Models (LLMs) are proficient at generating coherent and contextually relevant text but face challenges when addressing knowledge-intensive queries in domain-specific and factual question-answering tasks. Retrieval-augmented generation (RAG) systems mitigate this by incorporating external knowledge sources, such as structured knowledge graphs (KGs). However, LLMs often struggle to produce accurate answers despite access to KG-extracted information containing necessary facts. Our study investigates this dilemma by analyzing error patterns in existing KG-based RAG methods and identifying eight critical failure points. We observed that these errors predominantly occur due to insufficient focus on discerning the question's intent and adequately gathering relevant context from the knowledge graph facts. Drawing on this analysis, we propose the Mindful-RAG approach, a framework designed for intent-based and contextually aligned knowledge retrieval. This method explicitly targets the identified failures and offers improvements in the correctness and relevance of responses provided by LLMs, representing a significant step forward from existing methods. > [!Cite]- > Agrawal, Garima, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. “Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation.” arXiv:2407.12216. Preprint, arXiv, October 6, 2024. [https://doi.org/10.48550/arXiv.2407.12216](https://doi.org/10.48550/arXiv.2407.12216). > > [link](http://arxiv.org/abs/2407.12216) [online](http://zotero.org/users/17587716/items/C3WEGQW5) [local](zotero://select/library/items/C3WEGQW5) [pdf](file://C:\Users\erikt\Zotero\storage\L6S3AYTI\Agrawal%20et%20al.%20-%202024%20-%20Mindful-RAG%20A%20Study%20of%20Points%20of%20Failure%20in%20Retrieval%20Augmented%20Generation.pdf) ## Notes %% begin notes %% The authors propose Mindful-RAG, a framework designed for intent-based and contextually aligned knowledge retrieval. Extracts key entity relations from the sub-graph within one-hop distance MindfulRAG overcomes 2 limitations 1. Reasoning Failures: LLMs struggle to accurately interpret user queries and leverage contextual information, resulting in a misalignment between retrieved knowledge and query intent. 2. Structural Limitations: These failures primarily arise from insufficient attention to the structure of knowledge sources, such as knowledge graphs, and the use of inappropriate evaluation metrics. %% end notes %% %% begin annotations %% ### Imported: 2025-07-17 9:29 am Our investigation identified eight critical failure points in these systems, which we categorized into two primary areas: 1) Reasoning Failures: LLMs struggle to accurately interpret user queries and leverage contextual information, resulting in a misalignment between retrieved knowledge and query intent. 2) Structural Limitations: These failures primarily arise from insufficient attention to the structure of knowledge sources, such as knowledge graphs, and the use of inappropriate evaluation metrics. we propose Mindful-RAG, an approach that re-engineers the retrieval process to be more intent-driven and contextually aware. Mindful-RAG suggests to leverage the intrinsic parametric knowledge of LLMs to accurately discern the intent behind queries. This approach not only guides the retrieval process to ensure that the extracted context from the KG is relevant but also aligns it with the original intent of the query. Additionally, Mindful-RAG introduces advanced contextual alignment techniques for efficient knowledge graph navigation and incorporates a validation step to ensure the generated response meets the intended requirements. For our study, we chose the WebQuestionsSP (WebQSP) [7] dataset for knowledge graph question answering (KGQA), which is frequently utilized by KG-based RAG methods [8]. A review of response logs from both failed and successful interactions reveals that the LLM relies heavily on semantic matching. While this approach suffices for simple queries, it falls short in handling complex questions that demand multihop reasoning and deep contextual understanding. Therefore, improving intent identification and context alignment is essential for enhancing model performance. we introduce Mindful-RAG, designed to address two critical gaps: the lack of question intent identification and the insufficient contextual alignment with available knowledge. This approach employs a strategic hybrid method that integrates the model’s intrinsic parametric knowledge with non-parametric external knowledge from a KG. The following steps provide a detailed overview of our design and methodology, each accompanied by an illustrative example. Step 1. Identify key Entities and relevant Tokens: The first step is to pinpoint the key entities within a question to facilitate the extraction of pertinent information from an external KG or a sub-graph within a KG. Additionally, in our method, we task the LLM model with identifying other significant tokens that may be crucial for answering the question. For instance, consider the question from WebQSP, “Who is Niall Ferguson’s wife?” The key entity identified by the model is ‘Niall Ferguson’, and the other relevant token is ‘wife’. Step 2. Identify the Intent: In this step, we leverage the LLM’s understanding to discern the intent behind the question, prompting it to focus on keywords and phrases that clarify the depth and scope of the intent. For instance, in the provided example, the model identifies the question’s intent as “identify spouse”. Step 3. Identify the Context: Next, the model was instructed to understand and analyze the context of the question, which is essential for formulating an accurate response. For the provided example, the model identifies relevant contextual aspects such as “personal relationships,” “marital status,” and “current spouse.” Step 4. Candidate Relation Extraction: Next, the key entity relations are extracted from the sub-graph within one-hop distance. For our example, the candidate relations include information about the subject’s profession, personal life, and societal role. Step 5. Intent-Based Filtering and Contextual Ranking of Relations: In this step, the model conducts a detailed analysis to filter and rank the extracted relations and entities based on the question’s intent, ensuring relevance and accuracy. Relations are ranked according to their contextual significance, with the top-k relations being selected. For example, considering the intent and context in the given scenario, the model identifies “people.person.spouse s” as the most relevant relation. Step 6. Contextual Alignment of Constraints: In this step, the model considers temporal and geographical constraints by utilizing relevant data from various indicators to address more complex queries. This process ensures that responses are accurately tailored to specific times, locations, or historical periods. Step 7. Intent-Based Feedback: In the final step, the model is prompted to validate whether the final answer aligns with the initially identified intent and context of the question. If the answer does not meet these criteria, the model is instructed to revisit Step 5 and 6 to further refine its response. %% end annotations %% %% Import Date: 2025-07-17T09:29:50.070-06:00 %%