Retrieval-augmented generation (RAG) became the default pattern for enterprise search with language models. Chunk documents, embed them, retrieve the most similar passages, and let the model answer from them.
It works well for many questions. It struggles with others. When the answer depends on how things relate — which contracts reference this supplier, which systems depend on this service, which policies apply to this product in this region — similarity search over chunks often misses the connections.
That is where knowledge graphs, and the approach often called GraphRAG, come in.
Where plain RAG runs out
Vector retrieval finds passages that look like the question. That is a strength and a limit.
Common failure modes:
- Multi-hop questions — the answer needs facts from several documents linked by an entity
- Aggregation — “all”, “every”, “how many” questions need complete sets, not the top few chunks
- Global questions — themes or summaries across a whole corpus
- Ambiguous entities — the same name meaning different things in different contexts
- Relationships hidden in structure — ownership, hierarchy, dependency, sequence
Better chunking and re-ranking help at the margins. They do not give the system an explicit model of how things connect.
What a knowledge graph adds
A knowledge graph represents entities — people, products, systems, contracts, locations — and the relationships between them. Retrieval can then follow those links instead of relying only on textual similarity.
In a graph-based approach, the system typically:
- Extracts entities and relationships from documents, often with a language model
- Resolves duplicates so “the payments service” and its formal name are one node
- Links each fact back to the source passages that support it
- At query time, identifies relevant entities, traverses the graph, and retrieves the connected evidence
- Passes that structured context, with citations, to the model
Some variants also build summaries of clusters of related entities, which helps with broad, corpus-level questions.
It is not a replacement
GraphRAG is not a reason to throw away vector search. Most effective systems are hybrid:
- Vector search for direct, passage-level questions
- Graph traversal for relationship and multi-hop questions
- Keyword search for exact identifiers, codes and names
- A routing step or planner that decides which to use, or combines them
Choose based on the questions your users actually ask. Collect those questions in discovery before deciding on architecture.
The hard part is the graph
Building a useful graph is real data work.
- Schema design — which entity and relationship types matter for the use case, and which do not
- Extraction quality — model-based extraction makes mistakes, so sample and measure it
- Entity resolution — merging duplicates without merging things that only look alike
- Freshness — documents change, so the graph needs incremental updates, not a one-off build
- Access control — permissions on source documents must carry through to nodes, edges and summaries
Access control is easy to overlook. A relationship inferred from a restricted document can leak information even if the document itself is never shown.
Evaluating graph-based search
Evaluate retrieval and answers separately. Useful checks include:
- Does retrieval return the right entities and complete sets for aggregation questions?
- Are answers grounded in cited sources?
- How does accuracy compare with plain RAG on the same question set?
- What do extraction and update cost in time and compute?
If the graph does not beat vector search on your real questions, it is not worth the complexity.
Where to start
- Gather real user questions and tag the ones plain RAG answers poorly
- Define a small schema for one domain rather than modelling everything
- Build extraction with source links and sample its accuracy
- Carry document permissions through to the graph from the start
- Compare hybrid retrieval against your current system on the same evaluation set
Enterprise knowledge is connected. Search that understands the connections answers better questions.