By Ali Kore, Mahshid Alinoori, and Shayaan Mehdi, Vector Institute
Contents
- Introduction
- 1. One axis: retrieval becoming agentic
- 2. Two knobs, one frontier
- 3. Corpus, query, and what the substrate holds
- 4. What the published comparisons actually compare
- 5. What breaks when the graph is updated
- 6. What the agent knob changes
- 7. What we would measure before turning a knob
- References
Introduction
Retrieval-augmented generation now has a wide menu of upgrades. A reranker, a decomposed query, a knowledge graph, or an agent that takes retrieval into a loop are different kinds of change, with different costs.
One common way to order the options is as a ladder, starting with basic RAG, then advanced RAG, then possibly GraphRAG, then agents. When used as a decision procedure it suggests each step is an improvement on the last. Published comparisons do not generally support that reading.
A different axis is how much retrieval happens inside a reasoning loop, and, separately, what the retrieval substrate has to represent. A knowledge graph is one way to enrich that substrate. This post is a first pass at that framing, and, as an aside, what it would mean to keep a graph correct after the documents underneath it start to move.
1. One axis: retrieval becoming agentic
The axis we have been using is how much of our retrieval happens inside a reasoning loop.
Baseline RAG retrieves once and reads. A query is embedded, the nearest chunks come back, the generator answers. The pipeline is static and every request follows the same path.
Advanced RAG improves what that single pass returns. Hybrid lexical and dense search, cross-encoder reranking, contextual or hierarchical chunking, and query rewriting raise the quality of the candidate set without changing the control flow.
Agentic RAG moves part of retrieval into a loop. Query decomposition splits a question before retrieving; ReAct-style loops interleave reasoning with retrieval actions; corrective approaches such as CRAG judge what came back and retrieve again when it is insufficient.
Agent-native retrieval takes the vector index out of the loop. The agent navigates the corpus with search, file, and tool calls, in the way coding agents work over a repository. Retrieval reduces to actions the agent takes.
We treat these as positions on a spine. A static pipeline over a well-built index is a reasonable place to stay. Moving along the axis trades adaptivity for latency, token cost, and variance.
2. Two knobs, one frontier
The spine is easier to hold next to a second picture. Two overlapping circles: retrieval, the substrate a system can draw on, and agent, the loop reasoning over it. Moving along the spine grows the agent circle until, at the agent-native end, the two coincide.
Figure 1. Move the agent circle across retrieval, and set how much it can do. The overlap is how much retrieval happens inside the loop. The four named positions are readings of that overlap, not stops you are held to.
A working hypothesis is that quality can move by turning either knob. A richer substrate with a simple loop and a plain substrate with a sophisticated loop can land at similar answer quality with very different cost, latency, and operational shape.
Each knob has its own settings. The substrate knob is how the corpus is represented: flat chunks, hierarchical or contextual chunks, a hybrid lexical and dense index, a knowledge graph. All of these are built before a question arrives.
The agent knob is what the agent does with that substrate. At one end is a single shot: no loop at all, retrieve once and generate. Query decomposition splits the question and retrieves per part. A ReAct-style loop interleaves reasoning with retrieval actions and decides the next one from what the last returned. Self-critique adds a judgement step by scoring its own evidence or answer and retries when it falls short, which is the pattern CRAG and Self-RAG formalise.
Each of these is a different architecture with its own latency profile and its own failure modes.
Reranking belongs to neither knob. It is a stage that sits on top of whatever the substrate returns, which is why a graph plus a reranker is a normal thing to build. A knowledge graph with basic traversal and a flat chunk index with a capable agent are both designs where the relative complexity sits on either end.
Figure 2. Zoom into the overlap and it resolves into two wheels: how the corpus is represented, and what the agent does with it. Turn one and the other's reading does not move. One knob at a time is what makes a gain attributable.
If the two decisions collapse into one ladder, the expensive substrate (like a knowledge graph) is easy to buy first, because it sits higher on the imagined ordering. When evaluation falls short, changing one circle at a time makes it easier to find the root cause.
3. Corpus, query, and what the substrate holds
A second cut, orthogonal to the spine, is what the substrate has to represent. Two properties give a rough first map.
Corpus structure ranges from loose to dense. A loose corpus holds narrative or heterogeneous documents with sparse, unstable relationships between entities. A dense corpus has recurring entities, useful hierarchy, and relationships that stay meaningful across documents. Support tickets that carry product, fault, and resolution fields are denser than a set of unrelated essays.
Query type ranges from local fact retrieval to relationship-heavy reasoning. Local queries need one or two passages. Multi-hop queries combine facts across sources. Global queries ask about themes or coverage across a whole corpus. Path queries ask for the relationship chain itself, where the path is part of the answer.
Figure 3. A first map of corpus structure against query type, read off published task-level results. Operating constraints can move a cell.
On local factual queries over a loose corpus, a tuned retrieval stack is often enough. Graph extraction can add noisy relations without giving the query a capability it needs. On a dense corpus, metadata and entity filters sometimes capture enough structure for local questions that a graph would be premature.
Multi-hop is a closer call. Query decomposition and iterative retrieve-reason-retrieve loops work without a persistent graph. Graph methods look more plausible when the same relationships recur across traffic and the corpus is stable enough that construction might amortize. That is a bet.
Global sensemaking is a different workload again. Hierarchical summaries can cover a loose corpus. Community-based GraphRAG is interesting when the corpus supports stable communities and users keep asking corpus-wide questions. Explicit path queries are the cleanest motivation for a graph that we know, because the relationship path is part of the answer.
The grid is a starting map. Corpus size, update rate, latency budget, query mix, and whether anyone on the team has maintained a graph before can move a cell. A method that improves one benchmark can still be a poor fit if its index is rebuilt daily or its query path blows the latency budget.
4. What the published comparisons actually compare
Aggregate leaderboards flatten the conditional behaviour. Published comparisons show small gaps on some standard multi-hop tasks and large reversals on particular query types [1], [2].
| Task and metric | Retrieval baseline | Graph method | Difference |
|---|---|---|---|
| MuSiQue Recall@5 | NV-Embed-v2: 69.7 | HippoRAG 2: 74.7 | Graph +5.0 |
| 2Wiki Recall@5 | NV-Embed-v2: 76.5 | HippoRAG 2: 90.4 | Graph +13.9 |
| HotpotQA Recall@5 | NV-Embed-v2: 94.5 | HippoRAG 2: 96.3 | Graph +1.8 |
| Natural Questions F1 | Vector RAG: 68.18 | Community-GraphRAG Local: 65.44 | RAG +2.74 |
| MultiHop-RAG temporal | RAG: 25.73 | Community-GraphRAG Local: 49.06 | Graph +23.33 |
| MultiHop-RAG null | RAG: 91.36 | GraphRAG Global: 13.95 | RAG +77.41 |
Table 1. Selected published comparisons across different datasets and metrics. Recall@5 rows are from HippoRAG 2 [3]; the Natural Questions and MultiHop-RAG rows are from the RAG versus GraphRAG evaluation [2], whose GraphRAG configurations follow Edge et al. [4]. The rows use different tasks and scoring, so they do not combine into one ranking.
That incomparability is useful. On MultiHop-RAG temporal questions, the local community graph result exceeded the RAG result by 23.33 points. On null questions, where the right response is that the answer is absent, Global GraphRAG scored 13.95 against RAG's 91.36. A map-reduced answer over community summaries can produce a confident synthesis even when the source does not contain the answer.
Xiang et al. [1] report the same dependence on task and corpus structure. Their tests span loosely structured novels and densely structured oncology guidelines. RAG with a reranker led graph variants on Level 1 fact retrieval over the novel corpus. Graph methods improved Level 2 and Level 3 reasoning and summarization on the dense corpus. The paper also reports substantial prompt-token growth for some global GraphRAG configurations.
A benchmark set that reflected this would include local, multi-hop, temporal, global, and unanswerable questions in proportions that match expected traffic, and would record retrieval quality, answer correctness, latency, index cost, and update cost. Few published tables do all of that.
5. What breaks when the graph is updated
A vector index absorbs a revised document by re-embedding the changed chunks. A graph has no equivalent operation that is that cheap [5]. Extraction is probabilistic, so an update can change the topology while the index remains queryable. The failure, when it appears, looks like a wrong answer with a confident path.
Figure 4. One document changes. In a chunk index the change stays where it landed. In a graph it travels along whatever edges the extractor draws, and the nodes it reaches were never inspected.
We can enumerate at least three primary ways that this happens, though there are probably more.
Structural regression: re-ingesting a revised page can orphan a node, create a hub artifact, or drop the edges that hold a fault tree together. The graph still has a query API. A parameter that used to sit on a path now sits alone.
Semantic drift: entity typing, relation vocabulary, and co-reference shift between ingestion runs. The same object appears twice under two labels. A relation that was triggers last month is related_to this month. Traversal still runs. It follows the wrong name.
Completeness gaps: the graph can be connected and still miss the identifier, the triggering condition, or the resolution that made the path worth having. A fault links to a procedure and not to the parameter that causes it. Last week's answer accuracy will not be seen until someone asks that question.
Figure 5. One support graph, three ways an ingest can damage it. In each case the graph still has a query API and traversal still returns a path.
Answer-level evaluation scores the last question set. It does not inspect the graph that will serve the next one.
Figure 6. The graph laid out flat, with each measure as elevation. The surface turns on its own and cycles between the three; the cursor tilts it and lights the ridge under the pointer.
One stack we have been experimenting with runs three tiers over the graph itself, ordered from cheap and deterministic to expensive and generative.
Topology maps can flag isolation, degree skew, and relation-type collapse, and mark the high and low zones worth a look.
Constraint shapes are induced from a snapshot accepted as known-good and act as a fingerprint. They are silent on that snapshot by construction, so any new violation after an ingest is a candidate regression.
A third pass seeds an LLM on those zones and proposes an action: merge a split entity, split a collapsed hub, enrich an orphan, or prune a noisy bridge. Deterministic deltas and review items are easier to audit when they stay tagged separately.
A known-good snapshot can be versioned the way a schema is versioned. An update (a revised manual, a new article, a withdrawn page) could go through a gate before it is live for retrieval: topology delta, constraint delta, and a small held-out question smoke test. Findings would arrive as recommended actions. A human-in-the-loop could accept the merge, the split, or the enrichment. The new graph would not become the next known-good until that review closed, and only then would the fingerprint be re-induced.
Two practices look cheap enough to try before a graph is in front of users:
One is a synthetic setup where we add, revise, and withdraw documents on purpose and see whether the harness fires on the failure you planted.
The other is a written trigger. Deciding which events cause a re-ingest (a document revision, a schema change, a drift threshold on relation vocabulary) is an important consideration, which mirrors the way flat embedding tables update selectively.
This is a large part of what the substrate knob costs. On a dense, slowly changing corpus the gate looks feasible. On a corpus that moves every day it is a reason to leave the graph unbuilt and spend the effort on the other knob instead.
6. What the agent knob changes
Turning the other knob moves work from the index to the request path.
A simple agent can decompose a question, route local sub-questions to hybrid search, and use graph traversal only for relationship-heavy steps. A ReAct-style agent interleaves each retrieval action with a reasoning step over what the last one returned. A corrective loop in the CRAG mould scores the retrieved evidence and retrieves again when it falls short. A graph-oriented agent can inspect candidate paths, follow supporting edges, return to source spans, and stop when the evidence looks incomplete. An agent-native system navigates a structured corpus with search and document tools and no vector index at all.
In each case the substrate is one input to a policy. Agentic control will not repair a noisy graph, and a detailed graph will not decide when traversal is worth its latency. A comparison that mixed those two would be hard to read. One way to keep them apart is to run the same questions with a fixed generator, record every retrieval action, and compare the evidence delivered to the answer stage, not only the final text.
7. What we would measure before turning a knob
The questions we keep coming back to, before changing architecture, are ordinary ones.
When a question fails, was the right passage missing from the candidate set, or was it retrieved and unused?
If the evidence is missing, hybrid search, reranking, and better chunking are the cheap first moves. If the evidence is present and one pass cannot assemble it, decomposition and iterative retrieval are the next ones we would try.
A graph enters the conversation when explicit paths, entity aggregation, or corpus-wide structure still look necessary after that, and only if the corpus looks dense and stable enough to keep. Construction cost, query cost, and update cost belong in that conversation with answer quality.
References
- Zhishang Xiang et al. When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation, 2026 revision. arXiv:2506.05690. arxiv.org/abs/2506.05690
- Haoyu Han et al. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights, 2026 revision. arXiv:2502.11371. arxiv.org/abs/2502.11371
- Bernal Jimenez Gutierrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models, ICML 2025. arXiv:2502.14802. arxiv.org/abs/2502.14802
- Darren Edge et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization, 2024. arXiv:2404.16130. arxiv.org/abs/2404.16130
- Vector Institute. KG-RAG source and original article implementation, 2025. github.com/VectorInstitute/kg-rag
