Knowledge Graph-Informed Query Decomposition(KG-IQD): Hybrid KG-RAG Reasoning in Noisy Context
1National Institute of Technology, Patna
2IIIT Hyderabad
Abstract
Disaster data is inherently noisy and heterogeneous, posing significant challenges for automated reasoning. Retrieval-Augmented Generation (RAG) systems retrieve relevant documents but often fail to capture relational structure, impairing tasks such as comparative analysis and summarization, critical for disaster response. Knowledge graphs (KGs), while structured, are often sparse and lack contextual depth, which may not allow them to capture in-depth information needed for reasoning and planning queries. We propose Knowledge Graph-Informed Query Decomposition (KG-IQD), a hybrid KG-RAG framework that uses KGs to strategically guide the query decomposition into a set of focused, single-answer, contextually relevant subqueries, which perform the highly targeted RAG process, ensuring that the final LLM-generated answer is coherently grounded in both the retrieved textual evidence and the rich structured context of the KG. We evaluated KG-IQD against strong baselines using our novel Indian Disaster Dataset and an "LLM-as-judge" framework. To ensure a rigorous and fair evaluation, our methodology mitigates positional bias through output randomization and employs a metric that penalizes verbosity and irrelevance. The results demonstrate a clear advantage: KG-IQD outperforms RQ-RAG by 14% and KG by 18%. Our dataset, code, and output are publicly available at: github.com/hexronuspi/kg-iqd.
KG-IQD Framework
KG-IQD first grounds the user query by retrieving relevant triples from our disaster-specific KG. This structured context provides a semantically rich prior that guides an LLM in decomposing the initial complex prompt into a series of focused, single-answer sub-queries.
These sub-queries then drive a targeted RAG process that retrieves high-precision evidence from the raw text corpus. The final answer is synthesized by an LLM that integrates signals from both the retrieved textual context and the original KG grounding, enabling robust reasoning.
Specialized Dataset
We curated specialized corpora from publicly available, reputable sources, including government assessments, NGO reports, and academic studies. The corpus collected contains details of the 2015 Nepal Earthquake and the 2018 Kerala Flood, each having 43 top level schema and 207 sub-level.
Reasoning Queries
’A’: 2015 Nepal Quake, ’B’: 2018 Kerala Floods
- M1 (Medium): Which event had a greater impact on healthcare access: ’A’ or ’B’?
- M2 (Medium): Compare the environmental and socio-economic impacts between ’A’ and ’B’.
- H1 (High): What mitigation measures were most effective during ’A’?
- H2 (High): What logistical challenges affected evacuation efforts during ’B’?
Evaluation Method
We used a penalized LLM-as-Judge framework, randomizing outputs to avoid bias and rewarding high factual density.
| Scoring Criteria | Weight |
|---|---|
| Factual Density & Specificity | 60% |
| Efficiency & Clarity | 20% |
| Relevance & Topical Alignment | 20% |
Performance Scores
KG-IQD consistently outperforms all baselines, achieving a perfect score on all queries.
| Query ID | KG-Only | RQ-RAG | Base LLM | KG-IQD (Ours) |
|---|---|---|---|---|
| M1 | 9.3 | 7.5 | 4.5 | 10.0 |
| M2 | 8.0 | 9.0 | 6.0 | 10.0 |
| H1 | 9.4 | 9.7 | 3.8 | 10.0 |
| H2 | 7.0 | 9.0 | 5.0 | 10.0 |
References
- K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor, Freebase: A collaboratively created graph database for structuring human knowledge, in: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, ACM, 2008, pp. 1247–1250.[DOI]
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, in: Advances in Neural Information Processing Systems (NeurIPS), 2020.[URL]
- D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, J. Larson, From local to global: A graph rag approach to query-focused summarization, arXiv preprint arXiv:2404.16130 (2024).[URL]
- S. Ji, S. Pan, E. Cambria, P. Marttinen, P. S. Yu, A survey on knowledge graphs: Representation, acquisition, and applications, IEEE Transactions on Neural Networks and Learning Systems 33 (2021) 494–514.[URL]
- T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, A. Sabharwal, Decomposed prompting: A modular approach for solving complex tasks, in: International Conference on Learning Representations (ICLR), 2023.[URL]
- C.-M. Chan, C. Xu, R. Yuan, H. Luo, W. Xue, Y. Guo, J. Fu, Rq-rag: Learning to refine queries for retrieval-augmented generation, arXiv preprint arXiv:2404.00610 (2024).[URL]
- L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, Judging llm-as-a-judge with mt-bench and chatbot arena, arXiv preprint arXiv:2306.05685 (2023). NeurIPS 2023 Datasets and Benchmarks Track.[URL][DOI]