This presentation is going to be about retrieval augmented generation, or RAG, with Llama Index workflows. Um, yeah, a high-level overview of the agenda is I'm going to cover what RAG even is, uh why it's necessary, and some of the common techniques people have used to implement it.

I'm going to cover Llama Index workflows, which is a new addition to our core framework. Uh, show you guys Llama Cloud and what it can do and have a little short demo. Um, I can't really see the audience, but can I get a show of hands of who knows what RAG is? Oh my, okay. That I did not expect that many people, honestly. But uh, let's see. Oops. So I have a couple slides about what is RAG and the high level for those who didn't raise their hand is RAG stands for retrieval augmented

generation, and it's a technique that leverages custom data to enhance LLM output. I'm glad that you guys can see this diagram because it highlights exactly what RAG is. Given a simple user query, this is just like you asking a simple question to let's say a knowledge assistant or a chatbot. Um, we vectorize it and then we run some similar searches over a large corpus of data that you've ingested and shoved into your vector database. And it, that similar search will return some relevant contextual chunks that you can pass into your LLM prompt. Um, and then

the large language model and all its great power will return hopefully a very accurate response. However, as some of you might know who have implemented RAG systems, that there are some grave limitations, especially when trying to productionize these systems. Um, even though simple RAG has reduced the reduced

hallucinations, it hasn't gotten rid of them altogether, especially when there's more of a different variety of data sources, uh different types of documents, very complex queries, uh it's naive RAG tends to fall short. Um, and yeah, to the business, this results in limited time

savings as well as uh limited decision-making enhancement. So, not really good, uh chatGPT altogether. Yeah, so as we've come to realize over the last couple of years that chat uh complex use cases require much more data and much more advanced reasoning. Data comes from a variety of

data sources in different formats, and it's often times not a one-shot retrieval for all the contexts you need. To be able to uh retrieve from multiple different sources, you like understand if the data is correct or not, and yeah, these are some common limitations that

people run into. So, how can we go go beyond this simple RAG stack? Uh, in comes Agentic RAG. As many of you might know, the the world is becoming ever more agentic, um, yeah, as as Chi was going talking about, yeah, agents are going to be a core piece of a lot of gen

applications. So, Agentic RAG simply combines AI agents with with a variety of RAG techniques to enhance uh the end-to-end reasoning and output. So, as um as we try, as we begin to see more complex inputs, um complex questions, we can leverage the reasoning capability of LMs

to improve the overall query layer. So, some common common types of questions that inputs people see are summarization, research, multipart questions, and that just the naive naive RAG systems are quite incapable of handling. And we also want to generate more sophisticated forms of outputs, um a common two common

situations are uh report generation and form filling let's say I have um let's say I want to do a report a financial report over multiple industries through one prompt uh yeah how am I going to gather all the different types of information and give and generate a a

full report at the end um yeah so we've seen that there are a couple of common techniques that people can implement to u to create an a RAG system listed here we have query routing query decomposition structured data querying uh CAG and Heidi and I'll cover a few of

them that are more fundamental to agent RAG so the first one being query routing and the principle is quite simple you leverage an LLM to route a given input query to a variety of of different subquery tools um yeah as as shown in this diagram a simple example of query

routing could be given an LLM call can see it can query um a vector database or look at uh your chat history or existing memory um and we can continue to build on this this idea with query decomposition given um a a multi-part question for example the LLM can break this up into

different two simpler questions uh for example what is Uber's Revenue headcount the LLM can break it up into what is Uber's revenue and what is Uber's headcount and pass that into different function calls down the line and yeah I've listed a couple other

examples um more more complex examples another very common thing is structured data querying so uh naive RAG tends to the common imputation naive RAG is um vectorizing a ton of unstructured data or in documents and shoving it into some database but what if your data

lives in SQL databases or CSVs or XMLs when you kind of need a way to grab that data as well so I'm not sure many some of you might have heard of text to SQL but this is a common very specific implementation of this

technique yeah so how does one build an agentic racism this is the fun part guys we have a recent addition to our core framework is Llama Index Workflows um it is an event driven Asing first framework for building AI agents and it it really only comes with a few few building

blocks like these four primitives we have steps which are general which are just Python functions at the end of the day um we have events which uh help orchestrate your entire agent or your workflow uh we have tools which can be queuing tools retrieval tools or even just

function tools from other agents and and context which is another uh concept about memory uh one of the cool benefits of Llama Index Workflows is that it was built with productionizing in mind so we've also have another thing called Llama Deploy which helps you bring um

helps you have a control plan to orchestrate different agents together together or a single workflow and help orchestrate the entire thing in a dockerized way it also has integrations with uh observability tools like arise Phoenix you know shout out the hosts of this event and um yeah

workflows are automatically instrumented but yeah so now we have the agent how do we bring all the data together so to even have a functioning full like very powerful rack system it it requires a ton of a ton of data and we have a solution for that too uh

Something that the team has been working really hard on is Llama Cloud, and the aim of Llama Cloud is to take care of all the hard parts of RAG. Um, one of the major benefits is the large-scale data ingestion. So a lot of our customers have

massive, like S3 buckets or tons of files they want to they want to ingest. Llama Cloud takes care of it. Uh, we have a handful of different integrations as you can see listed for data sources. You can choose any embedding model provider: OpenAI, Azure, etc. And we also have um support

for a variety of data syncs. So, where, which kind of vector databases that you guys want uh your data to go into, tightly integrated with Llama Cloud is um Llama Parse. Uh, there are a lot of different um, I guess like data extraction tools out

there, but I think Llama Parse is the best. Uh, it supports a variety of file types. Yeah, over 100 of them, and excels particularly in um tables, charts, and and images. And yeah, this is a core piece of of of any RAG system, to be able to like

accurately get get get data from from your documents. And yeah, we also know that RAG isn't a black box, so we provide full customizability um in in in your retrieval system. So a variety of chunking strategies, hyperspace reranking, multimodal indexing, and so on

and so forth. Um, yeah, so we put these two pieces together, we get a gent RAG awesome. Um, and yeah, I got a quick demo for you all. So how do I like fire this up? Yeah, this one can we nice? Yeah. So here is a sneak peek into

Llama Cloud, it is currently only in beta. Uh, sorry for if you guys are on the wait list, we will be going GA pretty soon. And we have a couple simple concepts right now. We have a parsing page, which is a sandbox to use Llama.

Parse. Um, you can pass in any kind of complex data and and give it some some instructions and configuration. It'll get all the data for you. Uh, but the more powerful piece that I can show here is is the index creation workflow. Um, so as mentioned before, you can select any kind

of data source. We have even more data sources than just uh file storage types. You can import things from from like Confluence, Google Drive, and and more. Um, you can choose a handful of data syncs. Those might be interesting to some people here. Uh, you can choose any

embedding model provider. One that we released recently is multimodal indexing because a lot of documents have have images that they want to search over. Um, and you can do that very easily in LL Cloud. And obviously, parse parse settings, and you can set your own chunk settings

here. So you have full customizability over exactly how your data is processed and ingested with uh with Llama Cloud. And me go back here. So here's an example of of an index I created already. It includes a few data sources to even further enhance retrieval capabilities.

You can attach metadata to all your documents. It's a very common technique. We've seen a lot of a lot of people leverage. And you can also chat with it right out of the box. Yeah, nice. So we can take this and uh these indexes you can

query on them as if they're an API. Uh, here you can just, yeah, use this this library as offered by our framework and and some kind of query engine tool in in your workflows and yeah, so I have a notebook prepared for you guys. Oh, I

have to run through these steps again. Okay, but the the purpose of the notebook is to show you how to implement a very simple workflow and I've chosen to show a text-to-SQL example and just show how how easy it is to uh to implement with

with this product. Um, oh yeah, shout out to arise again. Yeah, so the first thing you want to do is create your retrievers and these uh these are other packages in in our Llama index framework. So for this example, we have a SQL database retriever, so we initialize our SQL database, load

it up with some with some sample data. Um, pass into a query engine, do the same thing for the Llama Cloud index and then we can create our tools. So uh yeah, I think people know what tools are and this description here is used as a selector by the LLM to understand where

to route a given query. And the next thing we do is create create our actual workflow. This is a very I guess basic example, but hopefully the the idea is is understood. Uh, we can create a handful of these different events and events can route between different steps. Uh, events

are just panic models, so you can attach different data to it if you want other downstream events um to to use some use some use some data from previous steps. Um, you can pass in LLM, you can pass in chat history, etc. And yeah, whenever for a step that has a

Start event, this indicates I guess the start of the workflow, um, and then it, the downstream steps, process uh a given chat query, and it can figure out exactly what uh what tool to call I guess in this step specifically. Yeah, so in just a few lines

we can create a fully functioning uh query router workflow and here are some example queries. So um given something that's more quantitative, as you can see here in the logs, it it'll do some work and figure out to to use the SQL Tool. Uh yeah, I don't want to spend time like

downloading the dependencies so I'm not going to run these, but uh essentially what will happen is if you provide a more qualitative question, it'll figure out that it needs to use the the other other tool, like the the Llama Cloud index tool. Um yeah, but we can take these

ideas and extend it even further for more complex complex use cases and systems. Uh for example, if I want to implement a human in the loop system, all you all somebody might need to do is add another event at the start or somewhere in some other step to create a loop that

somebody can, a human can intervene in. And yeah, this is we think that these Primitives are um the power of these Primitives are really felt in more more complex agentic systems

Agentic RAG with LlamaIndex Workflows

Arize AI is the watcher, troubleshooter and the guardrail on deployed AI.

