all right, so let's go and get started. My name is Jonathan Larsson, and today I'm excited to come here today to talk to you about GraphRAG. GraphRAG is an LLM-derived Knowledge Graph for RAG. And just to give you a small teaser, what you see over here on the right-hand side is actually a picture of Kevin Scott's behind the Tech podcast as an LLM-derived Knowledge Graph, an LLM memory representation. We'll be covering this in about the middle of the presentation here today, so just wanted to give you though a teaser of what's to come. So what is GraphRAG? GraphRAG is really a two-step process. It is first an indexing process that's run over top of private data to create LLM-derived knowledge graphs. These knowledge graphs serve as a form of an LLM memory representation, which can then be used by subsequent steps to then do better retrieval, which leads us to the Second Step. Second part of it is it's an LLM orchestration mechanism that utilizes those pre-built indices that I just talked about and

then those indices can be used to construct much much better more empowered rag operations and that's really leads us to the key differentiators of what graph rag allows us to do. The first is allows us to help enhance search relevancy. This is because it has a holistic view of the semantics across the entire data set. Second, it helps us enable new scenarios that would today require a very large context. For example, doing holistic data set analysis for trends, summarization, aggregation, things like that. So if you haven't had a chance to read it yet in the pre-add, encourage you to take a look at the blog post, take a look at the archive. They have a lot more technical details and measurements and evaluations in this space. But the one thing I want to ask of you today as we go through this presentation is how should we best drive impact with this technology. Okay, so let's go ahead and explore how graph rag actually works. The way that we explain graph rag is first to explain, of course, how Baseline rag.

works. So in Baseline rag, what you do is you take a private data set, you chunk it up using embeddings, and you store it into a vector database. Then you perform your neighbor search, and you can use those nearest neighbor searches to augment the context window. Graph rag is a parallel process to the way that Baseline rag works. What we do with this is we actually take the same text chunks, and then we take those sentences that are being extracted, and we ask the LLM to perform reasoning operations over the top of each sentence in a single pass through all of the data. So let's take this example sentence here: We have the P leader Sylvia Mar who took the stage with Luo Jack, founder of Sea of Our Wildlands. Now, the major differentiation here is you can see there's some named entity recognition done over the top of the sentence, and that's pretty typical for this type of text analysis. However, the major differentiation here is we're not just looking for the named entities; we're looking for the relationships between those entities and the strength of those.

relationships, and this is where gp4 really comes in to play a very strong leading role in the capability of this technology. For example, we can actually see that Sylvia Mark is very strongly related to the PO because she is the leader. We can see that she is perhaps weakly related to Save Our Wildlands because she is apparently taking the stage with its leader, but she's not the leader of that, and that's a major differentiation. Here is that gp4 can understand the semantics of these relationships, and that allows us to create weighted graphs from those relationships that are far richer than just like co-occurrence networks, which is where traditional ner would typically take this type of problem. So once we create these knowledge graphs, let's say, for example, we took all these sentences across this data set, we create a Knowledge Graph. Here what you get are a series of nodes that are connected to each other via these relationships, but that's not all we can do. Once we have the graph, we can utilize graph machine learning to do semantic aggregations and

heal Gomer over top of those structures. So, if you take this graph right here that we just created, which has no labeling on it, no colors if you will, we can then create a labeling on that at one level, and then we can hierarchically create subpartitions and subpartitions until you get down to individual nodes. This allows us effectively a granular filter that allows us to ask questions at any level of granularity across the data set for a semantic topic. Once that is built, we can then take that into a variety of different end use cases. Just to list out a couple, we could do data set question generation, we could do summarize Q&A, and a variety of other methods that we can talk about here later today. Um, so let's go ahead and jump into some demonstrations here and show you this technology in action. So, as I bring up the screen here, what you're going to see here are three different columns. In addition, I need to explain the data set that is behind each of these different columns that are going

we're going to be analyzing. So, we have three different rag systems that are implemented over top of one data set. This data set is about 3,000 articles in total. It's articles that originate from both the Russian and the Ukrainian side of the war of the conflict, and we're going to ask it a question that to all three different rag systems. We're going to ask it, what is Nov roia and what are its targets? There actually two questions. Now, if you're not familiar with no what Novar roia is, Novar roia are the uh Russian occupied portions of Ukraine. It's also a political movement, and then the water's targets are looking for what noia might be looking to destroy is basically. We know that there's some information inside this data set. Let's see how we can actually use rag-based retrieval to find those particular targets. So, using traditional Lang chain-based semantic search L is just using Baseline rag right here, we can see it actually fail to answer either of the two questions. Sometimes it can answer.

the first part of the question okay, but today it decided it couldn't come up with anything. The second column here is a much improved version of rag. This is, as people who've been using rag know, it oftentimes requires a lot of tuning, prompt engineering, and other improvements to actually make the rag work more effectively. And we can see that here. So, the left-hand side here is looking at Baseline rag. The middle column here is a much improved version. It's a supercharged version of rag. Now, if you read through this text right here, I'm just going to scroll through here real quick, you will actually see that it actually does an okay job of actually talking about the first part of this question: what is Novar roia, but it actually fails to mention anything about the specific targets that Novar roia are looking to destroy. In comparison, over here on the left-hand side, I'm going to highlight this first paragraph. This first paragraph here actually addresses the what is Novar roia part of the question.

that's good, but the second paragraph right here gives us a list of very highly specific targets that Novos was looking to destroy. For example, the National Television Company Ukraine, the radio station, the Canary. Looks like also some private private bank and Roshan properties. And they're also planning on terror attacks in the city of Odessa. And so, this is exactly what we're looking for. These are the specifics that we were seeing recall failures for on the Baseline Rag operations that we performed in the left two columns. Now when we perform Graph Rag, one of the nice features about it is it also allows us to actually look at the underlying Providence. So you'll see inside the text here, it refers to relationships. We can actually take a look at those relationships. I'm going to move to a second tool here, which is this is a purely Graph Rag tool. So I have two columns here. One is doing a local search, one is doing a community-based search. We'll get into the details of what each of these two are later. Both of

these answer the question correctly, but I'm going to focus on that question about Novar roia. Has targeted several entities for Destruction. Let's go ahead and click into those relationships. And if we open up the raw text, we can get the English translation because I can't read Russian or Ukrainian. And actually get to the originating text Chunk that was used to make the specific claim made above right here. So this can really help aid in understanding if there are hallucinations being made the system and detecting them and then also providing grounding and evidence, which is critical for analysts trying to use this data for their production purposes. The other thing we can do in here, of course, as well too, is we can use a second agent to really help reduce hallucinations by providing a verification score in this process. What we can actually do is we can also use the information that's provided in the context and the answer that was provided and use an independent agent to evaluate the two of those together to ask if anything was

hallucinated in those results, and so the score can really help provide an after-the-analysis as to whether the information was correctly grounded or not. Now let's go on to some of the capabilities where GraphRAG can show new opportunity spaces where regular RAG struggles. I'm going to ask it the question, What are the top five themes in the data? Now, when you ask this question to a typical Baseline RAG system, it's going to take this phrase, it's going to vectorize it using like an ADA embed, look for the nearest neighbors. The problem here is there's nothing in this query that would indicate any specific filtering over top of that data set, unless someone inside of this data set already wrote, What are the top five themes of the data, which is highly unlikely because each of these articles are independent of one another, and they don't have any knowledge of one another. So, what does Baseline R give us in this case? In this case, it gives us state of Russian economy. It actually did in this case also come up with one

example here about acknowledging that there seems to be a war the national rating the investment climate improving the quality of life and the meeting of Vladimir Putin. So I want to just take a step back for a second and kind of emphasize this is a data set that's primarily about a conflict. Like 80% of the articles are about a conflict, and you can see about 80% of the bullet points here don't have anything to do with the fact that there's a war going on. In contrast, because we can use the semantic and thematic and agative approaches that we've built over top of the graph machine learning parts of graph frag, we actually have a holistic understanding of what's happening in the data set. So on the graph rag side of things, you can see front and center the first thing it's talking about is the conflict and the military activity and how that plays through on each of the major themes that it then gives you in context. Now I do want to draw your attention to one other thing on this.

with Baseline rag, it's using about 5,000 tokens of context, and it returned in about 8 seconds. Graph rag here is a lot more expensive. It took about 50,000 tokens of context, and it took about 71 seconds to respond. But the important piece here is that though graph is using a lot more resources, it's providing a much, much richer and correct answer, which we've seen for our customers is the crucial piece and what they really care about, and they're willing to actually pay the extra cost to get these much better answers returned. Next, let's take a look at this data set from a network map perspective. So this is actually a visualization of the entire graph Network map of this data set, which is called the vena data set over the Russian Ukrainian War. Now again I said about 80% of it's on the war, so the main core of this is going to be about war topics. On the periphery, however, we'll find lots of normal topics, and one of the ones I like to look at in

particular is this group over here on the left. It turns out a lot of these entities here are about soccer, and you can see that they're semantically grouped next to each other in the embedded space, which is also great because it starts showing us that there are a group of entities that are co-related with one another. And then these colors here serve as a for us to actually, uh, look at the communities that define these semantic boundaries. So, for example, we can actually choose one of these colors. I'm going to choose a color for Community. Looks like number 450. I can extract the subgraph for it, and we can also extract out pregenerated reports that the LM has already generated on each of these Community structures. So, in this case here, we can see clearly this is the Novar Roosa community that was being used to answer some of those, um, queries that I made earlier in the demonstration. All right, let's go and switch context a little bit to a different data set. This is a data set where we took in all of

the transcripts from the behind the Tech podcast, which is of course hosted by Kevin Scott. So we can actually use the same type of methods here with craftag and ask it holistic, thematic, and Trend-type questions over top of this data set where, again, regular rag would tend to fail. So the first question I'm going to ask it are one of the top 10 technology Trends in the podcast. And so you can see here it has some good breath and diversity. We'll dive more into this here later too when we go to compare this with Gemini. So I'll ask you to, maybe, comment on some of these later because we'll actually be able to see it in side by side with Gemini. The next question, and this is an important one because we have a lot of other demonstrations tied to this one, are what are the most odd conversations discussed. Again, this is a question where, if you imagine Baseline rag being run over the top of this, it would perform very poorly. It would effectively pull back random.

chunks, and if those chunks had randomly picked up something with some odd conversations, it would comment on them, but it's not going to have the comprehensiveness and diversity that uh graph rack approach will. In this case, the last example I'm going to show you here is actually a side by side. So as I mentioned before, we have orchestration that runs over top of the indexes, over the indices, and we one. We have two of those methods that we're highlighting here today. And on this orchestration, we have one that does a local search of the knowledge graph. So it looks for like in this case nodes discuss artificial intelligence and looks at the nearest neighbors. Or we have a global search, which is going to be a more expensive operation that look at looks at the community summaries. And you can see the comparison of the depth and breadth between these two. So in this case here, you can see that very clearly the global summary here is providing you a much more comprehensive and diverse view than you are in like the local search. So let's take a look at Kevin's podcast, uh.

Knowledge Graph, an interactive graph visualization tool. Now, each of the nodes here represents an entity that the LLM extracted. And again, I just want to take you back for a second. This knowledge graph did not exist before the LLM was exposed to all the transcripts. It read through all those transcripts and created this from nothing, and that's pretty cool. So if we take a look at the colors here, the colors are semantic partitions that represent a highly topic. Now, one of the things I should mention, because I can only show one level of that hierarchy at a time, is that we're looking at the root hierarchy of this graph right now. So if we zoom into this node right here, let me go ahead and just pull it up here. We can see this node right here represents Kevin Scott. Zooming out and looking up a little bit, we can see that this node is Microsoft, has another very highly connected one, and another one right here, which is a major landmark, is Christina Wharton, who helps out on the podcast as.

well to now if we zoom into one of these colors, in particular, let's go take a look at this green section over here. One of the things we're going to notice is we can see all the entities that are being pulled out. So we can see things like RNA virus, Spike glycoprotein, SARS Corona virus 2, some things like synthetic biology, computational biology, and what's interesting here is it's actually grouped two episodes together into the same semantic topic. We have one with Drew Andy in episode 22, we have one with oh uh yeah one episode 33 of David Baker. I looked these two people up, it looks like they're both in the biology field so it totally makes sense. And it actually goes to show that graph is actually working in the sense that it pulled out all the entities and then actually grouped them semantically together which is exactly what we want because that will help us answer those questions. So if we get biology type questions on the orchestration side of the house, we can then come to this.

portion of the graph to then help better answer and augment those questions

GraphRAG: LLM-Derived Knowledge Graphs for RAG

Tech, AI, and Life from your resident Chaos coordinator.

Currently, I’m building next-generation AI products at Microsoft through the Office of the CTO with an emphasis on multimodal and large language models.

Over the last decade, I’ve successfully launched applied ML products across industries in health, gaming, finance, defense, energy, autonomous vehicles, semiconductors, and enterprise tech.

I’ve held product management roles at SambaNova Systems and C3.ai building out ML Platforms and deep learning offerings to enterprise customers. I continue to be an active ML practitioner and was the first data science hire at Uber’s Advanced Technologies Group working on building large scale computer vision and sensor fusion algorithms for self-driving cars.

Proud alum of UC Berkeley! Go Bears!


GraphRAG: LLM-Derived Knowledge Graphs for RAG

GraphRAG: LLM-Derived Knowledge Graphs for RAG

Scripts