로컬 모델을 사용하여 자체 반사 RAG 앱 구축하기

Lance가 오픈 소스 및 로컬 모델을 사용하여 검색, 채점 및 생성을 수행하는 자체 반영 RAG 앱을 구축하는 방법을 설명합니다.

### 오픈 소스 및 로컬 모델을 사용하여 자체 반사 RAG 앱 구축하기
* 🤖 Lance가 자신의 노트북에서 엄격하게 실행되는 **오픈 소스 및 로컬 모델**을 사용하여 자기 반성적인 **RAG 앱**을 구축하는 방법에 대해 설명합니다.
* 🔍 이 앱은 **질문을 기반으로 검색**을 수행하고, **자기 성찰, 추론, RAG 프로세스의 다양한 단계**를 다시 시도하는 과정을 포함합니다.
* 💻 AMA를 사용하여 **로컬 LLM을 실행**하고 다양한 노드와 **조건부 에지가 있는 그래프**로 논리적 워크플로우를 구현하는 방법을 시연합니다.
* 📊 구현에는 **그래프 상태를 정의**하고, **각 노드와 조건부 가장자리에 대한 함수**를 만들고, **그래프 구성을 배치**하는 작업이 포함됩니다.
* 🧠 Lance는 그래프를 컴파일하고 앱의 기능을 테스트하여 **검색, 채점, 웹 검색 및 생성 단계의 로컬 실행**을 보여줍니다.
* 📈 그는 **복잡한 논리적 추론 작업에 로컬 모델**을 사용하는 것과 **논리적 흐름을 제한하는 신뢰성**에 대해 강조합니다.

hi, this is Lance from Lang chain team. I'm going to talk about building a self-reflective rag apps from scratch using only open source and local models. Um, that run strictly on my laptop. Now, one of the most interesting trends in the rag research and a lot of like methods that become pretty popular in recent U months and weeks is this idea of self-reflection. So, when you do rag, you perform retrieval based upon a question from an index, and this idea of self-reflection is saying based upon, for example, the relevance of the retriev documents to my question, or based upon, you know, the quality the generations relative to my question, or the generations relative to the documents. I want to make. I want to perform some kind of reasoning and potentially feedback and retry various steps. So, that's kind of the big idea, and there's a few really interesting papers that implement this, and what I want to kind of show is that implementing these ideas using something that we've developed recently called Langra is a really nice approach. Um.

it works really well with local LLMs that are much smaller for example than you know API uh gated very large-scale Foundation models um and so we're going to look at a particular paper called corrective rag or C rag now this paper is kind of um there's been some attention on for example Twitter about this work uh it's a really neat paper and the idea is actually pretty simple and straightforward if you go down to the figure here you're going to do perform retrieval and you're going to grade the documents relative to the question so you're kind of doing a relevance grading and there's some theistic like basically if the documents are deemed correct they actually do some uh knowledge refinement where they further strip the documents to compress relevant chunks within the documents and retain them um and if the documents are either deemed ambiguous relative to the query or incorrect it performs a web search and supplements retrieval with the Webster so that's kind of the big idea but it's a nice illustration of this General principle

of don't just do rag as kind of like a you know a single-shot process where you perform retrieval and then go to generation you can actually perform self-reflection and reasoning. You can retry. You can uh retrieve from alternative sources and so forth. That's kind of the big idea. Now in our build here we're going to make some minor simplifications. Um here's kind of a layout of the graph that we're interested in. We're going to perform retrieval and for that we're going to use no embeddings which run locally. Um we're going to build a node for grading those documents relative to the question to say, are they relevant or not. And if any documents are deemed irrelevant, we'll go ahead and do a query rewrite web search. And we'll s go ahead to generation based upon the web search results. So that's the flow. Now first things first is how do I get started running LMS locally and and kind of where do I go and where I often direct people and what I found to be really useful is

AMA um it is a really nice way to run models locally uh for example on your Mac laptop very easily, and they are launching support for various other platforms as well um. So basically if you go to their website, it's very simple. You simply download their application, um you can see it's running here on my machine, um and once you have it downloaded, all you need to do is you can go to their model list, and you can kind of search around. So you can actually look, I think it's sorted by popularity, so you can see mraw obviously a really interesting open source model, um is kind of one of the top. So you can see it has like 210,000 polls. It's one of the top models. I click on this, and where this takes me is a model page. I can look at this tags, uh Tab, and this basically shows me, um a bunch of model versions that I can really easily, uh just like download and run, and we'll, we'll show how to do that here very.

shortly um what I'm going to do is I'm going to choose mrol instruct so that is their 7 billion uh parameter instruct model and so all I would do I'm going to go over to my notebook here so I have an empty notebook and all I've done is I've already uh done a few pip installs and I've also set uh a few environment variables to use Langs Smith and we'll see why that's useful later that's really all I've done now I'm going to put a note here to uh for olama and what I'm going to do is this AMA pull the model I want and you just run that so normally this will take a little bit because you're actually pulling the model and typically it's like a couple gigs I actually already have this model so it's faster um it's actually already done but that's really all you do okay so that's kind of like step one and then what we're going to do is I'm just going

to create this variable local llm um that I am going to yeah so I'm just going to Define this variable Mr all instruct because this is the model that I download using a llama pull miston struct that's all that's going on here so this is be the llm I'm going to work with I've pulled this so it's local on my system it's available V Lama which is basically running in the background on my system and you can see is really seamless and easy to use now the first thing I want to do for this approach is I'm going to call this um index so because this was a corrective rag approach I need an index that I care about that I'm actually performing rag on and so here I'm going to use uh a particular blog post that I like on agents and we can like pull it up here and have a look let pull it up over here actually so this is a pretty neat blog post on autonomous agents it's like pretty long and mey so it's kind of like

a good target for performing retrieval on lots of details here. Really neat, really detailed blog post. So what I'm going to do is I'm going to load it here, I'm going to split it, and I'm going to use a chunk size of 500 tokens. Um, these are kind of somewhat arbitrary parameters, you can play with these as you want. The point is here I'm just building a quick local index. Um, so I load it, I split it into chunks. Now, this is the interesting bit. I'm going to use GPT for all embeddings from nomic, which is, let's actually pull up the link here. I had it available here. So these are, um, you can see right here, it is a CPU, uh, CPU-optimized contrastively trained, basically, eser model. Um, so you can drill into sentence Transformers. So you can see, um, yep, so there it is. The initial work is described in our paper, espert, basically. So the key point is this, this is a locally running CPU-optimized embedding model that works quite well, I found.

runs on your system no API nothing so it's pretty nice. Runs fast so we're going to go ahead and use that. Um from our friends at Nomic. And I'm also going to use Chroma, which is an open-source local Vector store that's really easy to spin up. Runs locally and all I'm doing is I'm taking my documents. Um I'm going to define a new collection. Taking my embedding model GPD for all embeddings. I'm going to create a retriever from this. So there we go, okay. It shows you some, uh, you some parameters. So cool. I have a retri so we can actually call. We can say get relevant documents. Um and I can say something about like. Um let's say like agent memory or something, you know, let's just test and okay cool. Look at that. So it's nice and quick. We get a bunch of documents out that relate to memory. So yeah, you can see memory stream like the documents are are sayane. So it looks like everything's kind of working here. So that's great. We have a

retriever. Now, let's think a little bit about what we want to do next. So, when I do these kinds of logical workflows as graphs, first I always try to lay out the logic. And let me move this up here. I try to lay out the logical steps. And in each logical step, what's happening is I'm transforming state. So, in these graphs, really all you're doing is you're defining a state that you're just modifying throughout the flow of the graph. Now, in this case, because we're interested in RAG, our state is just going to be a dictionary. And that dictionary, you can see I actually kind of schematically laid it out here, it's just going to contain a few keys that are things relevant to RAG. It's going to be like a question, then it's going to be you pen documents to your dict, and then eventually you're independent generation. So, that's really all that's going on in terms of like how

your State's being propagated through the graph, and at every node, you're making some modification to State. That's the key point. So you're basically going to do you start with a question from the user, you perform retrieval relevant to the question. Um, you're then going to grade the documents, so you're going to do a modification of the documents. Then you're going to make a decision, are they relevant or not? If they're not relevant, um, you're going to transform the query, so you modify the question, do a web search. The final step is a generation based upon the D documents. So that's your flow. Now what I want to call out here is there's one very important, what we call conditional Edge, where depending upon the results of the grading step, I want to do one thing or another. So I'm going to make a decision. So I want to show you something that's very convenient, um, that we can use with olama to help us here. So this is, I'm going to kind of make a note here, um, to note what I'm going to highlight.

so this is AMA Json mode. So the basic logic of that conditional Edge decide to generate is going to be something like this. I already have this prompt laid out, um, but it's basically going to be: I'm going to take a document and I'm going to take my question, and I'm going to do some kind of comparison to say, is the document relevant to the question? That's really what I want to do. But here's the catch, because I want that edge to process very particular output, either yes or no, I want to make sure that my output is structured in a way that can reliably be interpreted downstream in my graph. This is where Json mode from Alama is really useful, and you can see all I do is now I'm importing chat llama. This is going to reference that local model that I specified up here, Mr. Instruct, which I've downloaded. So I have the model locally, and I'm just flagging this, um, format Json, to tell the model to output Json specifically. And what I'm going to do in

my prompt here. I'm basically saying, you know, you're a grer. Um, here's the documents, here's the question, and here's the catch. Give a binary score: yes, no. Um, and provide it as Json with a single key score and no Preamble, no explanation. So I kind of explain in the prompt what I want, and when I call this with Json mode, uh, it will enforce that Json is returned, and hopefully with this single key, we expect score and either binary yes, no. And when I'm going to run that as a chain, so I'm going to supply that prompt to my llm, and I'm going to then parse that Json string out into a Json object, which I can work with. So let's try that. We're going to try to run this chain. We defined. We're going to run retrieval on here's a here's a question, here's our docs. Let's grade one of the docs using basically passing question and one document, and we're going to take the page content from the document, which is like basically all the text, and we're

going to run this. So let's test that quickly. And it is still running. Now it's finished. Let's check the output. Here we can see we get a Json back, which just is the score, yes, no. So that's exactly right. That's what we want. And we can actually look under the hood here at um, yeah. So we can actually look under the hood in Langs Smith at that grading process. And we can see here that our prompt got populated with um, the context. So here is the document, um, and um, right here was a question. Here is a document, and um, the task was of course to grade it. So we can see here's like the full prompt. You're a grader assessing the relevance retri document. Here's a document. And then here's the model output score, yes. So this is really nice. We've basically enforced the output from our local llm, um, using Json mode. So we know every time it's going to output binary yes, no score as a Json object, which we again extract. So that's a very key point that

I just wanted to flag, it's a very nice thing that AMA offers. That's extremely helpful when building out, particularly these kind of logical graphs, where you really want to constrain the flow at certain edges. So that's kind of the like really key thing I wanted to highlight. A lot of the rest of this is actually pretty straightforward. So let's now define our graph state. This is the dictionary that we're going to basically pass between our nodes. So this is just some code I'm going to copy over. This is defining your graph state. You're just saying it's a dict. That's all there's really to that. Now, here is where I'm going to copy over some code that basically implements a function for every node and every conditional edge in our graph. So if you remember, we can kind of go over and look. Our graph is laid out like this, and all we're doing is for every node drawn, we're going to find a corresponding function here which performs some operation. So retrieve is basically just doing we had our retri

defined get relevant documents and write them out to state so again we take a question in so if you look here we basically have this state dict passed into the function. We extract the state dict here. We extract the question from the state dict. We do retrieval, and we write that state dict back out to the. So you think about every node is just doing some modification on the state, reading it in, doing something, writing it back out. That's really all that's going on, and we can really just march across our little like diagram here and see how um basically each one of these nodes is implemented as a function. And again, you can see in every case we're using uh for example cadow llama. In some of these cases, we don't need Json mode. So if we're just doing like a generation step, um as you can see here, we don't need Json mode for the grading we do. So we're actually going to implement here the same thing we just showed um chat AMA using Json mode. And what's going to happen is we can see we generate our

score every time, and then we can extract our grade from that JSON, and then we know the grade is going to be constrained to the output, yes or no. Then here's the key point: we do some logical reasoning on that, um, to say, for example, if the grade is yes, um, then we're going to, um, like append the document if it's relevant. If not, then what we're going to do is we're going to filter that document out, and we're also going to set this flag to search, perform web search, as yes. So what's really happening here is we are kind of applying a kind of a logical gate to say if any document is scored as relevant, then we just add it to our final list of filtered documents. If not, we're going to go ahead and do a web search, and we're going to set the search flag to be yes, and we're not going to include that document in the output. And you can see here we return a dictionary which contains our filtered documents, our question, and then that flag to run web.

search yes or no. You can see it was default no, but if we ever encounter an irrelevant document, we change that to yes. So that's really all that's going going on here. Um, you can see we do our query transform down here. Again, we just use um, Mr. all again here is like a transform prompt, but you kind of get the idea. Um, web search node, we use Tav web search here. It's really kind of a nice quick way to perform web searches. Um, and you can see we just supplement the documents with the web search results. And then this was kind of the final step where we wrote out yes or no to our search key. And depending upon the state, which we can read in here, we make decision to uh, basically either return transform query or return generate, which will basically that's determining the next uh, node to go to. Um, so this decide to generate is our conditional edge that's actually right here. And so it's looking at the results that we wrote out from grade documents, in particular that uh, search yes or no key in our dict.

and it's then going to basically determine the next node to traverse to. That's really all we're doing here, so that's kind of nice. Now what we're going to do is we kind of copied over all these um, these functions. We then can go ahead and run that, and now we just lay out our graph. So again, our graph was kind of explained here, and here's where we actually just lay out the full kind of graph organization, um, how we're going to connect each node. So we add the nodes first, we set our entry point, and then we add the edges accordingly between the nodes. And basically the logic here just maps over to our diagram here. That's really all that's happening, um, cool. So I'm going to go ahead and go down, and now let's kind of see this all working together. So I'm going to go ahead and compile my graph, and I'm going to go ahead and ask a question. Explain how the different types of agent memory work. And what I'm going to do, let's go back to our D.

diagram so we can kind of reference that. I'm going to call this, and I'm actually just going to like this will like Traverse every step along the away, and it'll print out something to explain what's happening so you can see. I perform retrieval, and now I'm doing my grading steps, and this is all running locally. Um, and they were all deemed relevant, so then I'm going to go ahead and generate, and it's running right now, and there we go. So we can go over to Lang Smith, and let's actually have a look at what happened under the hood. So this is what just ran. So we can see that at each one of these steps, we called Shadow llama with our mraw 7B model that's running locally. Um, and this is our grading step, so this was each document being graded. Um, so again, like look at this, so it outputs a binary score yes no as a dict, that's great. Um, so this has a bunch more down here, so these are all of our documents, uh, graded, and now here is that.

final LLM call, which basically packed all that into our RAG prompt. You're an assistant for question-answering task. Use the following to answer the question. Here's all of our docs. Here's the answer. So that's pretty cool. Um, we can see that this multi-step logical flow all works. Um, now let's try something kind of interesting. I'm going to ask a question that I know is not in the context and see if it will kind of perform that default to do web search. So, um, I'm going to say, Explain how Alpha Codium works. So this is a recent paper that came out that's not relevant at all to this blog post. So I know that retrieval should not be considered relevant. And let's go ahead and run that and convince ourselves that that's true. So good, this is perfect. So the grader is determining these documents are not relevant, and so it should be making that decision to perform web search. So it should be kind of going to this lower branch, transform the query, run web search, and

looks like that all ran so it tells us Alpha coding is an open-source AI coding generation tool developed by Cod M. This is perfect, that's exactly what it is, and we can actually go into Langs Smith and again see what happened here. So you can see here the trace is a little bit more extensive because all of our grades are incorrect so or irrelevant. Again, we get the nice Json out um and okay so this is pretty cool. So this was our question rewriting node so basically providing an improved input question without any Preamble. So what is the mechanism behind Alpha codium functionality? So it modifies the question we use Tali search right here so it basically does retrieval, it searches for Stuff related to Alpha codium so that's great and then we finally passed that to our model for Generation based on this new context and there we go Alpha codom Source AI code assistant tool um so that kind of gives you the main idea and the key point is this is all running locally again. I used GPT for all embeddings for indexing up.

at the top right here, and I used AMA with mrol 7B instruct um, and Json mode for that one crucial step where I need to constrain the output to be kind of a score of yes, no um, and for other things. I just use the model without Json mode to perform Generations like to question rewrite or to do the final generation. So in any case, I hope this gives you kind of an overview of how to think about building logical uh flows. Doesn't have to be rag, but rag is a really good kind of use case uh for this using local models and Lang graph. And the thing I want to kind of leave you with is there is a lot of interest in complex logical reasoning using local llms and a lot of you know focus on using agents. And I do want to kind of encourage you to think about depending on the problem you're trying to solve. You may or may not actually need an agent. It's possible that kind of implementing a state machine or a graph kind of as shown here with some series.

of logical steps. This can incorporate cycles or loops back to like prior stages. We have some more complex examples that show that um, this actually can work really well with local mod models because a local model is only performing a step um within each node. So, you're kind of constraining it to like just do this little thing, just do this little thing, like just rewrite the question, just grade the document, rather than using the local LLM um as like, you know, an agent executor that has to make all these decisions kind of jointly um or kind of in a less controlled workflow where, for example, like the ordering of these various tasks can be determined arbitrarily by the agent. Here we really nicely constrain the logical flow and let the local model just do little tasks at each step. And I've just found it to be a lot more reliable and really useful for these kinds of like logical reasoning tasks um. So, hopefully this is helpful. Give it a try um and we'll make sure all this code is easily shared. Thank you. Thank

Building Corrective RAG from scratch with open-source, local LLMs

Learn more about building LLM applications with LangChain

LangChain

오픈 소스 및 로컬 모델을 사용하여 자체 반사 RAG 앱 구축하기

자기 성찰은 검색 기반 질문 답변 프로세스에서 추론과 피드백을 가능하게 하는 최근 RAG 연구의 인기 트렌드입니다.

Corrective RAG 논문에서 볼 수 있듯이 Langra와 로컬 LLM을 사용하여 이러한 아이디어를 구현하면 대규모 기초 모델보다 더 접근하기 쉬운 접근 방식을 제공합니다.

앱의 워크플로에는 로컬 임베딩을 사용한 검색, 관련성에 대한 문서 채점, 관련 없는 문서에 대한 웹 검색 수행, 웹 검색 결과에 기반한 생성 등이 포함됩니다.

AMA로 모델을 실행하는 쉽고 원활한 프로세스

AMA는 컴퓨터에서 로컬로 모델을 실행하는 간단한 방법을 제공하며, 다른 플랫폼에 대한 지원도 곧 추가될 예정입니다.

애플리케이션을 다운로드하고 목록에서 모델을 선택하면 시스템에서 쉽게 가져와 실행할 수 있습니다.

이 프로세스는 원활하고 효율적이어서 특정 인덱스에 대한 래그 수행과 같은 작업에 선택한 모델을 로컬에서 사용할 수 있습니다.

문서 검색에 GPT 사용하기: 단계별 가이드

로컬 인덱스는 문서 임베딩을 위해 GPT를, 벡터 저장소를 위해 크로마를 사용해 구축됩니다.

검색 프로세스에는 관련 키와 문서 조작을 포함한 RAG용 상태 사전을 정의하는 작업이 포함됩니다.

논리적 워크플로는 프로세스 전반에 걸쳐 상태를 변환하는 일련의 단계로 배치되어 있습니다.

그래프 처리에서 조건부 에지 로직에 Json 모드 사용

핵심 프로세스는 관련 정보를 검색하고, 문서를 채점하고, 질문과의 관련성에 따라 결정을 내리는 것입니다.

조건부 에지 로직에 대해 예 또는 아니오의 이진 점수로 출력의 신뢰할 수 있는 해석을 보장하기 위해 Alama의 Json 모드가 사용됩니다.

Json 모드를 사용하면 단일 키 점수로 Json을 반환하므로 그래프에서 추가 처리를 위해 Json 객체로 쉽게 파싱할 수 있습니다.

랭스 스미스의 채점 프로세스 분석하기

랭스 스미스는 채점 과정에 일관된 이진(예/아니오) 채점 출력을 적용하기 위해 Json 모드를 사용하여 명확하고 구조화된 결과를 제공합니다.

그래프 상태 사전을 사용하면 그래프의 각 노드에 대한 함수를 구현할 수 있으므로 각 노드의 특정 작업에 따라 상태를 쉽게 수정할 수 있습니다.

Json 모드의 구현은 채점 프로세스에서 특히 중요하며, 평가를 위한 표준화된 출력을 보장합니다.

논리적 추론 및 문서 필터링 프로세스

문서는 논리적 추론을 통해 포함 여부를 결정하기 위해 JSON 데이터를 기반으로 점수를 매기고 등급을 매깁니다.

문서가 관련성이 있다고 평가되면 최종 목록에 추가되고, 그렇지 않으면 필터링되어 웹 검색이 실행됩니다.

이 프로세스는 필터링된 문서와 웹 검색을 수행하기 위한 플래그를 포함하는 사전을 생성합니다.

질문 답변을 위한 다단계 논리 흐름의 로컬 작업

검색, 채점 및 생성을 위해 로컬 GPT 모델을 사용하는 질문 답변의 다단계 논리적 흐름이 시연됩니다.

이 시스템은 질문이 문맥과 관련이 없는 경우 웹 검색을 기본값으로 설정하여 개선된 입력 질문을 제공하고 새로운 문맥에 따라 답변을 생성하는 데 성공합니다.

이 프로세스는 모든 임베딩에 대해 GPT로 구동되며 완전히 로컬에서 실행됩니다.

AMA와 mrol 7B 지시어 및 Json 모드를 함께 사용하면 특정 요구에 맞게 출력을 제한하는 데 도움이 될 수 있습니다.

RAG는 의사 결정을 위해 항상 에이전트가 필요하지 않고 로컬 모델과 Lang 그래프를 사용하여 논리적으로 추론하는 데 유용한 사용 사례입니다.

논리적 단계가 있는 상태 머신이나 그래프를 구현하면 로컬 모델과 잘 작동하여 논리적 추론 작업에 보다 안정적인 접근 방식을 제공할 수 있습니다.